[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:39:20 dist-test-slave-vkjp krb5kdc[32418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:39:20 dist-test-slave-vkjp krb5kdc[32418](info): set up 2 sockets
Aug 06 04:39:20 dist-test-slave-vkjp krb5kdc[32418](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 04:39:21.290078 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:39:21 dist-test-slave-vkjp krb5kdc[32418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:39:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:39:21.430971 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:36179
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:36179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:21.544992 32432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:21.545280 32432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:21.545322 32432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:21.549345 32432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:21.549404 32432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:21.549423 32432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:21.549448 32432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:21.549463 32432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:39:21.554683 32432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:36179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:36179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:21.555815 32432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:21.556851 32432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:21.566072 32438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:21.566033 32437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:21.566239 32432 server_base.cc:1034] running on GCE node
W20260806 04:39:21.567137 32440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:21.567675 32432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:21.568770 32432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:21.569964 32432 hybrid_clock.cc:648] HybridClock initialized: now 1785991161569924 us; error 53 us; skew 500 ppm
Aug 06 04:39:21 dist-test-slave-vkjp krb5kdc[32418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991161, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:21.572846 32432 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:39:21.573765 32432 webserver.cc:460] Webserver started at http://127.31.166.190:43453/ using document root <none> and password file <none>
I20260806 04:39:21.574369 32432 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:21.574457 32432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:21.574692 32432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:21.577041 32432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "5c4d440d5859483f852f59a09510d6e2"
format_stamp: "Formatted at 2026-08-06 04:39:21 on dist-test-slave-vkjp"
server_key: "ea6e18a855ed58c6501cac569a65972f"
server_key_iv: "6cf5b190bc61d2ebda8a32d9d011d9e5"
server_key_version: "encryptionkey@0"
I20260806 04:39:21.577884 32432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "5c4d440d5859483f852f59a09510d6e2"
format_stamp: "Formatted at 2026-08-06 04:39:21 on dist-test-slave-vkjp"
server_key: "ea6e18a855ed58c6501cac569a65972f"
server_key_iv: "6cf5b190bc61d2ebda8a32d9d011d9e5"
server_key_version: "encryptionkey@0"
I20260806 04:39:21.581826 32432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 04:39:21.584564 32448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:21.585975 32432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:39:21.586112 32432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "5c4d440d5859483f852f59a09510d6e2"
format_stamp: "Formatted at 2026-08-06 04:39:21 on dist-test-slave-vkjp"
server_key: "ea6e18a855ed58c6501cac569a65972f"
server_key_iv: "6cf5b190bc61d2ebda8a32d9d011d9e5"
server_key_version: "encryptionkey@0"
I20260806 04:39:21.586217 32432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:21.612540 32432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:21.616675 32432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:21.616863 32432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:21.625379 32432 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:36179
I20260806 04:39:21.625408 32502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:36179 every 8 connection(s)
I20260806 04:39:21.626543 32432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:21.628014 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 32432
I20260806 04:39:21.628152 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:21.629689 32410 external_mini_cluster.cc:1428] Setting key c04432827fc772ec7a36867cb04fbd05
I20260806 04:39:21.630549 32503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:21.638288 32503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2: Bootstrap starting.
I20260806 04:39:21.641211 32503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:21.642228 32503 log.cc:824] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:21.644512 32503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2: No bootstrap required, opened a new log
I20260806 04:39:21.649694 32503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4d440d5859483f852f59a09510d6e2" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36179 } }
I20260806 04:39:21.650098 32503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:21.650184 32503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c4d440d5859483f852f59a09510d6e2, State: Initialized, Role: FOLLOWER
I20260806 04:39:21.650729 32503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4d440d5859483f852f59a09510d6e2" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36179 } }
Aug 06 04:39:21 dist-test-slave-vkjp krb5kdc[32418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:21.650915 32503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:21.650996 32503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:21.651155 32503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:21.652360 32503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4d440d5859483f852f59a09510d6e2" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36179 } }
I20260806 04:39:21.652782 32503 leader_election.cc:302] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c4d440d5859483f852f59a09510d6e2; no voters:
I20260806 04:39:21.653252 32503 leader_election.cc:288] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:21.653352 32508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:21.653620 32508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c4d440d5859483f852f59a09510d6e2, State: Running, Role: LEADER
I20260806 04:39:21.654019 32508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4d440d5859483f852f59a09510d6e2" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36179 } }
I20260806 04:39:21.654666 32503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:21.655268 32510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c4d440d5859483f852f59a09510d6e2. Latest consensus state: current_term: 1 leader_uuid: "5c4d440d5859483f852f59a09510d6e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4d440d5859483f852f59a09510d6e2" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36179 } } }
I20260806 04:39:21.655486 32510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:21.655771 32509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c4d440d5859483f852f59a09510d6e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c4d440d5859483f852f59a09510d6e2" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36179 } } }
I20260806 04:39:21.655962 32509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:21.656011 32514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:21.658627 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:21.633436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57846 (local address 127.31.166.190:36179)
0806 04:39:21.634081 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:21.634103 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:21.635238 (+ 1135us) server_negotiation.cc:407] Connection header received
0806 04:39:21.639555 (+ 4317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:21.639572 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:21.639898 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:21.640183 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:21.643289 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:21.644277 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:21.645968 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:21.646409 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:21.646547 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:21.651838 (+ 5291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:21.651845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:21.651860 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:21.651894 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:21.653997 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:21.654696 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:21.654703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:21.654708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:21.654881 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:21.655386 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:21.655391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:21.655394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:21.656077 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:21.656276 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:21.657391 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0806 04:39:21.657688 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":231,"threads_started":1}
I20260806 04:39:21.659554 32514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:21.665766 32514 catalog_manager.cc:1347] Generated new cluster ID: 32b8eab14730439e84fbe373857c1560
I20260806 04:39:21.665844 32514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:21.674249 32514 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:21.675392 32514 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:21.681495 32514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c4d440d5859483f852f59a09510d6e2: Generated new TSK 0
I20260806 04:39:21.682211 32514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:39:21.719695 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36179
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:21.856086 32532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:21.856338 32532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:21.856383 32532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:21.860610 32532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:21.860671 32532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:21.860782 32532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:39:21.866053 32532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36179
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:21.867293 32532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:21.868386 32532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:21.878453 32540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:21.878525 32537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:21.878533 32538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:21.878867 32532 server_base.cc:1034] running on GCE node
I20260806 04:39:21.879323 32532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:21.880044 32532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:21.881232 32532 hybrid_clock.cc:648] HybridClock initialized: now 1785991161881197 us; error 45 us; skew 500 ppm
Aug 06 04:39:21 dist-test-slave-vkjp krb5kdc[32418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991161, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:21.884096 32532 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:39:21.885001 32532 webserver.cc:460] Webserver started at http://127.31.166.129:34839/ using document root <none> and password file <none>
I20260806 04:39:21.885586 32532 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:21.885680 32532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:21.885923 32532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:21.888131 32532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "937ca0f115df44428d756e03becfee24"
format_stamp: "Formatted at 2026-08-06 04:39:21 on dist-test-slave-vkjp"
server_key: "a04846403da58947eec0b458c795404a"
server_key_iv: "5f7cddb2baeb498977640461636a334d"
server_key_version: "encryptionkey@0"
I20260806 04:39:21.888724 32532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "937ca0f115df44428d756e03becfee24"
format_stamp: "Formatted at 2026-08-06 04:39:21 on dist-test-slave-vkjp"
server_key: "a04846403da58947eec0b458c795404a"
server_key_iv: "5f7cddb2baeb498977640461636a334d"
server_key_version: "encryptionkey@0"
I20260806 04:39:21.892760 32532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:39:21.895272 32548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:21.896523 32532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:21.896662 32532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "937ca0f115df44428d756e03becfee24"
format_stamp: "Formatted at 2026-08-06 04:39:21 on dist-test-slave-vkjp"
server_key: "a04846403da58947eec0b458c795404a"
server_key_iv: "5f7cddb2baeb498977640461636a334d"
server_key_version: "encryptionkey@0"
I20260806 04:39:21.896790 32532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:21.920454 32532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:21.924316 32532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:21.924575 32532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:21.925235 32532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:21.926129 32532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:21.926201 32532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:21.926275 32532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:21.926318 32532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:21.935802 32532 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:38729
I20260806 04:39:21.935906 32672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:38729 every 8 connection(s)
I20260806 04:39:21.936751 32532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:39:21.937634 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 32532
I20260806 04:39:21.937747 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:39:21.938076 32410 external_mini_cluster.cc:1428] Setting key 8a626c6a178fa36dc4ea9e72edbf6a60
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
Aug 06 04:39:21 dist-test-slave-vkjp krb5kdc[32418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991161, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:21.957829 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:21.938966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:50115 (local address 127.31.166.190:36179)
0806 04:39:21.939137 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:21.939141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:21.941756 (+ 2615us) server_negotiation.cc:407] Connection header received
0806 04:39:21.943145 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:21.943156 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:21.943215 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:21.943407 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:21.944738 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:21.945322 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:21.946550 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:21.946814 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:21.946941 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:21.952836 (+ 5895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:21.952839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:21.952842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:21.952873 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:21.954625 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:21.955493 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:21.955497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:21.955499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:21.955655 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:21.956145 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:21.956152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:21.956156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:21.956453 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:21.956567 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:21.957451 (+ 884us) server_negotiation.cc:299] Negotiation successful
0806 04:39:21.957608 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:39:21.958806 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:21.939368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36179 (local address 127.31.166.129:50115)
0806 04:39:21.941551 (+ 2183us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:21.941599 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:21.942846 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:21.943622 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:21.943630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:21.944021 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:21.944556 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:21.944571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:21.945474 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:21.945480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:21.946401 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:21.946411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:21.946995 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:21.946999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:21.947221 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:21.947973 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:21.947991 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:21.952583 (+ 4592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:21.954859 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:21.954867 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:21.954888 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:21.955361 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:21.955782 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:21.955788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:21.955790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:21.955980 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:21.956589 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:21.956597 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:21.957089 (+ 492us) client_negotiation.cc:769] Sending connection context
0806 04:39:21.957382 (+ 293us) client_negotiation.cc:240] Negotiation successful
0806 04:39:21.957766 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1974,"thread_start_us":244,"threads_started":1}
I20260806 04:39:21.960296 32673 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36179
I20260806 04:39:21.960574 32673 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:21.961148 32673 heartbeater.cc:511] Master 127.31.166.190:36179 requested a full tablet report, sending...
I20260806 04:39:21.962932 32466 ts_manager.cc:194] Registered new tserver with Master: 937ca0f115df44428d756e03becfee24 (127.31.166.129:38729)
I20260806 04:39:21.964190 32466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:50115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:39:21.970767 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36179
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:22.084848 32679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:22.085108 32679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:22.085151 32679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:22.089190 32679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:22.089250 32679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:22.089313 32679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:39:22.094714 32679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36179
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:22.095829 32679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:22.096789 32679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:39:22.106698 32679 server_base.cc:1034] running on GCE node
W20260806 04:39:22.106710 32685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:22.106647 32686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:22.106647 32688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:22.107439 32679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:22.108076 32679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:22.109262 32679 hybrid_clock.cc:648] HybridClock initialized: now 1785991162109255 us; error 48 us; skew 500 ppm
Aug 06 04:39:22 dist-test-slave-vkjp krb5kdc[32418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991162, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:22.112437 32679 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:39:22.113350 32679 webserver.cc:460] Webserver started at http://127.31.166.130:36439/ using document root <none> and password file <none>
I20260806 04:39:22.113919 32679 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:22.114022 32679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:22.114251 32679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:22.116328 32679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "4b6ca2e562fd4ce98a0df9bb91a155f6"
format_stamp: "Formatted at 2026-08-06 04:39:22 on dist-test-slave-vkjp"
server_key: "eecd675ce35b6806599ea88671666f66"
server_key_iv: "32e0dba2246ebfc433eae4c41837be5c"
server_key_version: "encryptionkey@0"
I20260806 04:39:22.116895 32679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "4b6ca2e562fd4ce98a0df9bb91a155f6"
format_stamp: "Formatted at 2026-08-06 04:39:22 on dist-test-slave-vkjp"
server_key: "eecd675ce35b6806599ea88671666f66"
server_key_iv: "32e0dba2246ebfc433eae4c41837be5c"
server_key_version: "encryptionkey@0"
I20260806 04:39:22.120560 32679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:39:22.122937 32695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:22.124158 32679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:22.124296 32679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "4b6ca2e562fd4ce98a0df9bb91a155f6"
format_stamp: "Formatted at 2026-08-06 04:39:22 on dist-test-slave-vkjp"
server_key: "eecd675ce35b6806599ea88671666f66"
server_key_iv: "32e0dba2246ebfc433eae4c41837be5c"
server_key_version: "encryptionkey@0"
I20260806 04:39:22.124414 32679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:22.143355 32679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:22.146764 32679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:22.146988 32679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:22.147641 32679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:22.148486 32679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:22.148558 32679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:22.148636 32679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:22.148679 32679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:22.158277 32679 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:35817
I20260806 04:39:22.158322 345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:35817 every 8 connection(s)
I20260806 04:39:22.159296 32679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:39:22.168390 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 32679
I20260806 04:39:22.168505 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:39:22.168818 32410 external_mini_cluster.cc:1428] Setting key c4e74d76c971422c73b482ac5b4c454c
Aug 06 04:39:22 dist-test-slave-vkjp krb5kdc[32418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991162, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:22.173847 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:22.161220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:59899 (local address 127.31.166.190:36179)
0806 04:39:22.161355 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:22.161359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:22.162138 (+ 779us) server_negotiation.cc:407] Connection header received
0806 04:39:22.163646 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:22.163650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:22.163704 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:22.163780 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:22.164933 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:22.165476 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:22.166687 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:22.166981 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:22.167188 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:22.169994 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:22.169997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:22.169999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:22.170025 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:22.171750 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:22.172304 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:22.172312 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:22.172313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:22.172391 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:22.172737 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:22.172742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:22.172744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:22.172933 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:22.173040 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:22.173542 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 04:39:22.173675 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:39:22.174755 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:22.161544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36179 (local address 127.31.166.130:59899)
0806 04:39:22.161978 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:22.162022 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:22.163443 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:22.163941 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:22.163948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:22.164296 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:22.164790 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:22.164803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:22.165661 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:22.165665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:22.166546 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:22.166555 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:22.167198 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:22.167202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:22.167386 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:22.167972 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:22.167989 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:22.169813 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:22.171866 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:22.171871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:22.171884 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:22.172190 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:22.172490 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:22.172493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:22.172495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:22.172637 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:22.173041 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:22.173048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:22.173294 (+ 246us) client_negotiation.cc:769] Sending connection context
0806 04:39:22.173488 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 04:39:22.173814 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":116,"threads_started":1}
I20260806 04:39:22.176021 346 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36179
I20260806 04:39:22.176304 346 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:22.176931 346 heartbeater.cc:511] Master 127.31.166.190:36179 requested a full tablet report, sending...
I20260806 04:39:22.178225 32466 ts_manager.cc:194] Registered new tserver with Master: 4b6ca2e562fd4ce98a0df9bb91a155f6 (127.31.166.130:35817)
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:39:22.179123 32466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:59899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:39:22.198292 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36179
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:34211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:22.313447 351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:22.313733 351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:22.313779 351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:22.317888 351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:22.317951 351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:22.318046 351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:39:22.323311 351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:34211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36179
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:22.324389 351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:22.325443 351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:22.335351 360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:22.335435 357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:22.335358 356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:22.335569 351 server_base.cc:1034] running on GCE node
I20260806 04:39:22.335973 351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:22.336599 351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:22.337757 351 hybrid_clock.cc:648] HybridClock initialized: now 1785991162337725 us; error 44 us; skew 500 ppm
Aug 06 04:39:22 dist-test-slave-vkjp krb5kdc[32418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991162, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:22.345803 351 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:39:22.346724 351 webserver.cc:460] Webserver started at http://127.31.166.131:37877/ using document root <none> and password file <none>
I20260806 04:39:22.347360 351 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:22.347425 351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:22.347695 351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:22.349756 351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "c95e77969d6a45739b15d03c4ea9a8e4"
format_stamp: "Formatted at 2026-08-06 04:39:22 on dist-test-slave-vkjp"
server_key: "9b451e0edf9e2f27071304a67e6f4e20"
server_key_iv: "5c93abdab0b670aef9cf54ad5d140461"
server_key_version: "encryptionkey@0"
I20260806 04:39:22.350328 351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "c95e77969d6a45739b15d03c4ea9a8e4"
format_stamp: "Formatted at 2026-08-06 04:39:22 on dist-test-slave-vkjp"
server_key: "9b451e0edf9e2f27071304a67e6f4e20"
server_key_iv: "5c93abdab0b670aef9cf54ad5d140461"
server_key_version: "encryptionkey@0"
I20260806 04:39:22.354008 351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 04:39:22.356671 367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:22.357898 351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:39:22.358022 351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "c95e77969d6a45739b15d03c4ea9a8e4"
format_stamp: "Formatted at 2026-08-06 04:39:22 on dist-test-slave-vkjp"
server_key: "9b451e0edf9e2f27071304a67e6f4e20"
server_key_iv: "5c93abdab0b670aef9cf54ad5d140461"
server_key_version: "encryptionkey@0"
I20260806 04:39:22.358129 351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:22.381986 351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:22.385074 351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:22.385275 351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:22.385939 351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:22.386790 351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:22.386837 351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:22.386876 351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:22.386894 351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:22.396463 351 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:45275
I20260806 04:39:22.396486 491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:45275 every 8 connection(s)
I20260806 04:39:22.397327 351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:39:22.405411 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 351
I20260806 04:39:22.405537 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:39:22.405807 32410 external_mini_cluster.cc:1428] Setting key b16f3424f5b4050d2d392e8c5445640a
Aug 06 04:39:22 dist-test-slave-vkjp krb5kdc[32418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991162, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:22.411124 32505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:22.399225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:48517 (local address 127.31.166.190:36179)
0806 04:39:22.399411 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:22.399416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:22.400201 (+ 785us) server_negotiation.cc:407] Connection header received
0806 04:39:22.401563 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:22.401566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:22.401629 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:22.401737 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:22.402914 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:22.403533 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:22.404646 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:22.404877 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:22.404987 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:22.407560 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:22.407563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:22.407566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:22.407592 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:22.409131 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:22.409645 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:22.409649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:22.409650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:22.409731 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:22.410076 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:22.410079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:22.410081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:22.410255 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:22.410372 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:22.410830 (+ 458us) server_negotiation.cc:299] Negotiation successful
0806 04:39:22.410943 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:39:22.411876 494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:22.399625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36179 (local address 127.31.166.131:48517)
0806 04:39:22.400061 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:22.400101 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:22.401359 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:22.401919 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:22.401926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:22.402303 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:22.402761 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:22.402773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:22.403662 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:22.403670 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:22.404519 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:22.404529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:22.405026 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:22.405031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:22.405189 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:22.405835 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:22.405851 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:22.407404 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:22.409267 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:22.409272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:22.409284 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:22.409536 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:22.409832 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:22.409835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:22.409837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:22.409973 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:22.410372 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:22.410379 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:22.410605 (+ 226us) client_negotiation.cc:769] Sending connection context
0806 04:39:22.410786 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 04:39:22.411115 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":121,"threads_started":1}
I20260806 04:39:22.412923 492 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36179
I20260806 04:39:22.413112 492 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:22.413507 492 heartbeater.cc:511] Master 127.31.166.190:36179 requested a full tablet report, sending...
I20260806 04:39:22.414453 32466 ts_manager.cc:194] Registered new tserver with Master: c95e77969d6a45739b15d03c4ea9a8e4 (127.31.166.131:45275)
I20260806 04:39:22.415009 32466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:48517
I20260806 04:39:22.420023 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 04:39:22 dist-test-slave-vkjp krb5kdc[32418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991162, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 04:39:22 dist-test-slave-vkjp krb5kdc[32418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991162, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.166.129@KRBTEST.COM
I20260806 04:39:22.446242 500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:22.436828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36966 (local address 127.31.166.129:38729)
0806 04:39:22.437115 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:22.437128 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:22.437152 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:39:22.437201 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:22.437206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:22.437380 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:22.437507 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:22.438191 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:22.438968 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:22.439970 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:22.440331 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:22.440417 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:22.442991 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:22.443000 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:22.443013 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:22.443037 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:22.444568 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:22.444994 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:22.444998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:22.445000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:22.445080 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:22.445367 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:22.445369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:22.445370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:22.445595 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:22.445765 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:22.445963 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 04:39:22.446085 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":63,"threads_started":1}
W20260806 04:39:22.447353 32579 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36966
Aug 06 04:39:22 dist-test-slave-vkjp krb5kdc[32418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 04:39:22 dist-test-slave-vkjp krb5kdc[32418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.129@KRBTEST.COM
I20260806 04:39:22.472469 500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:22.464606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.31.166.129:38729)
0806 04:39:22.464830 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:22.464834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:22.464857 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:39:22.465022 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:22.465025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:22.465073 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:22.465150 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:22.465786 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:22.466307 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:22.467147 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:22.467393 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:22.467476 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:22.469758 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:22.469761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:22.469763 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:22.469786 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:22.471051 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:22.471483 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:22.471486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:22.471487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:22.471562 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:22.471791 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:22.471793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:22.471794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:22.471953 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:22.472049 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:22.472249 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 04:39:22.472344 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 04:39:22.473563 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 32532
I20260806 04:39:22.480127 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 32679
I20260806 04:39:22.487550 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 351
I20260806 04:39:22.494218 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 32432
2026-08-06T04:39:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2349 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:39:22 dist-test-slave-vkjp krb5kdc[508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:39:22 dist-test-slave-vkjp krb5kdc[508](info): set up 2 sockets
Aug 06 04:39:22 dist-test-slave-vkjp krb5kdc[508](info): commencing operation
krb5kdc: starting...
W20260806 04:39:23.573496 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:39:23 dist-test-slave-vkjp krb5kdc[508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:39:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:39:23.695976 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:39837
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:37977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:39837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:23.810034 522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:23.810326 522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:23.810369 522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:23.814527 522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:23.814595 522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:23.814620 522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:23.814637 522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:23.814653 522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:39:23.820222 522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:37977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:39837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:39837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:23.821310 522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:23.822432 522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:23.831429 528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:23.831509 527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:23.831697 530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:23.831744 522 server_base.cc:1034] running on GCE node
I20260806 04:39:23.832374 522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:23.833421 522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:23.834597 522 hybrid_clock.cc:648] HybridClock initialized: now 1785991163834568 us; error 41 us; skew 500 ppm
Aug 06 04:39:23 dist-test-slave-vkjp krb5kdc[508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991163, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:23.837502 522 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:39:23.838420 522 webserver.cc:460] Webserver started at http://127.31.166.190:41515/ using document root <none> and password file <none>
I20260806 04:39:23.838981 522 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:23.839076 522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:23.839345 522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:23.841361 522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "a75e0b275f744ee7bb7b51e627895d3e"
format_stamp: "Formatted at 2026-08-06 04:39:23 on dist-test-slave-vkjp"
server_key: "bd727356a53d5b6d6fc6c168e31d59dc"
server_key_iv: "6c7c397ea7ccfb38b7e9d8f1d3ce36a5"
server_key_version: "encryptionkey@0"
I20260806 04:39:23.841926 522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "a75e0b275f744ee7bb7b51e627895d3e"
format_stamp: "Formatted at 2026-08-06 04:39:23 on dist-test-slave-vkjp"
server_key: "bd727356a53d5b6d6fc6c168e31d59dc"
server_key_iv: "6c7c397ea7ccfb38b7e9d8f1d3ce36a5"
server_key_version: "encryptionkey@0"
I20260806 04:39:23.845803 522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 04:39:23.848394 537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:23.849611 522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:39:23.849753 522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "a75e0b275f744ee7bb7b51e627895d3e"
format_stamp: "Formatted at 2026-08-06 04:39:23 on dist-test-slave-vkjp"
server_key: "bd727356a53d5b6d6fc6c168e31d59dc"
server_key_iv: "6c7c397ea7ccfb38b7e9d8f1d3ce36a5"
server_key_version: "encryptionkey@0"
I20260806 04:39:23.849888 522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:23.872255 522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:23.875272 522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:23.875483 522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:23.882761 522 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:39837
I20260806 04:39:23.882783 593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:39837 every 8 connection(s)
I20260806 04:39:23.883706 522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:23.886421 594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:23.891759 594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e: Bootstrap starting.
I20260806 04:39:23.892675 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 522
I20260806 04:39:23.892791 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:23.893069 32410 external_mini_cluster.cc:1428] Setting key 9758597c8f17714745eceb42c93773f6
I20260806 04:39:23.894622 594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:23.895398 594 log.cc:824] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:23.897536 594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e: No bootstrap required, opened a new log
I20260806 04:39:23.901448 594 raft_consensus.cc:348] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75e0b275f744ee7bb7b51e627895d3e" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39837 } }
I20260806 04:39:23.901693 594 raft_consensus.cc:374] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:23.901782 594 raft_consensus.cc:729] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a75e0b275f744ee7bb7b51e627895d3e, State: Initialized, Role: FOLLOWER
Aug 06 04:39:23 dist-test-slave-vkjp krb5kdc[508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:23.902249 594 consensus_queue.cc:260] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75e0b275f744ee7bb7b51e627895d3e" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39837 } }
I20260806 04:39:23.902402 594 raft_consensus.cc:388] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:23.902494 594 raft_consensus.cc:482] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:23.902594 594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:23.903755 594 raft_consensus.cc:504] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75e0b275f744ee7bb7b51e627895d3e" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39837 } }
I20260806 04:39:23.904124 594 leader_election.cc:302] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a75e0b275f744ee7bb7b51e627895d3e; no voters:
I20260806 04:39:23.904469 594 leader_election.cc:288] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:23.904544 599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:23.904736 599 raft_consensus.cc:686] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [term 1 LEADER]: Becoming Leader. State: Replica: a75e0b275f744ee7bb7b51e627895d3e, State: Running, Role: LEADER
I20260806 04:39:23.905022 599 consensus_queue.cc:237] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75e0b275f744ee7bb7b51e627895d3e" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39837 } }
I20260806 04:39:23.905728 594 sys_catalog.cc:564] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:23.906219 601 sys_catalog.cc:455] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a75e0b275f744ee7bb7b51e627895d3e. Latest consensus state: current_term: 1 leader_uuid: "a75e0b275f744ee7bb7b51e627895d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75e0b275f744ee7bb7b51e627895d3e" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39837 } } }
I20260806 04:39:23.906350 601 sys_catalog.cc:458] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:23.907037 600 sys_catalog.cc:455] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a75e0b275f744ee7bb7b51e627895d3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a75e0b275f744ee7bb7b51e627895d3e" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39837 } } }
I20260806 04:39:23.907336 600 sys_catalog.cc:458] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:23.908273 605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:23.908653 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:23.894187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.31.166.190:39837)
0806 04:39:23.894598 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:23.894613 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:23.894647 (+ 34us) server_negotiation.cc:407] Connection header received
0806 04:39:23.895407 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:23.895425 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:23.895798 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:23.896070 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:23.896802 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:23.898225 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:23.899112 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:23.899800 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:23.899968 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:23.902624 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:23.902630 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:23.902650 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:23.902676 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:23.904716 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:23.905305 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:23.905310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:23.905315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:23.905482 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:23.905814 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:23.905819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:23.905823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:23.906380 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:23.906552 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:23.907335 (+ 783us) server_negotiation.cc:299] Negotiation successful
0806 04:39:23.907632 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":135,"threads_started":1}
I20260806 04:39:23.911587 605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:23.916922 605 catalog_manager.cc:1347] Generated new cluster ID: 57933a1425e849eb9bd9d1a042389f11
I20260806 04:39:23.916983 605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:23.923623 605 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:23.924554 605 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:23.930402 605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a75e0b275f744ee7bb7b51e627895d3e: Generated new TSK 0
I20260806 04:39:23.931056 605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:39:23.961731 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39837
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:37977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:24.090219 621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:24.090521 621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:24.090602 621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:24.094713 621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:24.094815 621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:24.094911 621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:39:24.100467 621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:37977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39837
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:24.101586 621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:24.102653 621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:24.112329 627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:24.112278 626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:24.112417 621 server_base.cc:1034] running on GCE node
W20260806 04:39:24.112600 629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:24.113000 621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:24.113605 621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:24.114768 621 hybrid_clock.cc:648] HybridClock initialized: now 1785991164114751 us; error 29 us; skew 500 ppm
Aug 06 04:39:24 dist-test-slave-vkjp krb5kdc[508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991164, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:24.117625 621 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:39:24.118554 621 webserver.cc:460] Webserver started at http://127.31.166.129:41119/ using document root <none> and password file <none>
I20260806 04:39:24.119158 621 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:24.119222 621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:24.119443 621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:24.121459 621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "af8ff74281ab4e9eb7ffcc02b65ee890"
format_stamp: "Formatted at 2026-08-06 04:39:24 on dist-test-slave-vkjp"
server_key: "54b768cda801cce09b24df5b40f3d5d6"
server_key_iv: "2f24cb6dbf553f29997201b17a07f634"
server_key_version: "encryptionkey@0"
I20260806 04:39:24.122030 621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "af8ff74281ab4e9eb7ffcc02b65ee890"
format_stamp: "Formatted at 2026-08-06 04:39:24 on dist-test-slave-vkjp"
server_key: "54b768cda801cce09b24df5b40f3d5d6"
server_key_iv: "2f24cb6dbf553f29997201b17a07f634"
server_key_version: "encryptionkey@0"
I20260806 04:39:24.125741 621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:39:24.128266 636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:24.129491 621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:39:24.129624 621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "af8ff74281ab4e9eb7ffcc02b65ee890"
format_stamp: "Formatted at 2026-08-06 04:39:24 on dist-test-slave-vkjp"
server_key: "54b768cda801cce09b24df5b40f3d5d6"
server_key_iv: "2f24cb6dbf553f29997201b17a07f634"
server_key_version: "encryptionkey@0"
I20260806 04:39:24.129736 621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:24.159999 621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:24.163400 621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:24.163616 621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:24.164237 621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:24.165066 621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:24.165136 621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:24.165207 621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:24.165241 621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:24.174144 621 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:43205
I20260806 04:39:24.174216 755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:43205 every 8 connection(s)
I20260806 04:39:24.175056 621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:39:24.179068 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 621
I20260806 04:39:24.179241 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:39:24.179503 32410 external_mini_cluster.cc:1428] Setting key 7e9d42e7822be6cab10ef5716ad9fffc
Aug 06 04:39:24 dist-test-slave-vkjp krb5kdc[508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991164, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
I20260806 04:39:24.191538 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.176840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:42073 (local address 127.31.166.190:39837)
0806 04:39:24.176998 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:24.177004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:24.177896 (+ 892us) server_negotiation.cc:407] Connection header received
0806 04:39:24.179668 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:24.179672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:24.179730 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:24.179827 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:24.181249 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.182217 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.183507 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.183920 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.184078 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.187333 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:24.187337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:24.187339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:24.187365 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:24.189032 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:24.189720 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:24.189724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:24.189726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:24.189818 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:24.190240 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:24.190244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:24.190246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:24.190452 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:24.190571 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:24.191188 (+ 617us) server_negotiation.cc:299] Negotiation successful
0806 04:39:24.191340 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 04:39:24.192435 758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.177258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39837 (local address 127.31.166.129:42073)
0806 04:39:24.177726 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:24.177770 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:24.179404 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:24.180031 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:24.180042 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:24.180480 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:24.181094 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.181110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.182379 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.182382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.183379 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.183389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.184123 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.184127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.184384 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.185083 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:24.185111 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:24.187075 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:24.189170 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:24.189178 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:24.189198 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:24.189591 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:24.189928 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:24.189931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:24.189933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:24.190124 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:24.190570 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:24.190578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:24.190878 (+ 300us) client_negotiation.cc:769] Sending connection context
0806 04:39:24.191127 (+ 249us) client_negotiation.cc:240] Negotiation successful
0806 04:39:24.191482 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
I20260806 04:39:24.193964 756 heartbeater.cc:347] Connected to a master server at 127.31.166.190:39837
I20260806 04:39:24.194226 756 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:24.194808 756 heartbeater.cc:511] Master 127.31.166.190:39837 requested a full tablet report, sending...
I20260806 04:39:24.196568 555 ts_manager.cc:194] Registered new tserver with Master: af8ff74281ab4e9eb7ffcc02b65ee890 (127.31.166.129:43205)
I20260806 04:39:24.197700 555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:42073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:39:24.210299 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39837
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:37977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:24.319936 761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:24.320199 761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:24.320243 761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:24.324558 761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:24.324617 761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:24.324679 761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:39:24.330032 761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:37977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39837
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:24.331122 761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:24.332142 761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:24.342026 768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:24.342134 761 server_base.cc:1034] running on GCE node
W20260806 04:39:24.342020 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:24.342020 766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:24.342665 761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:24.343413 761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:24.344607 761 hybrid_clock.cc:648] HybridClock initialized: now 1785991164344591 us; error 26 us; skew 500 ppm
Aug 06 04:39:24 dist-test-slave-vkjp krb5kdc[508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991164, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:24.347453 761 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:39:24.348340 761 webserver.cc:460] Webserver started at http://127.31.166.130:41657/ using document root <none> and password file <none>
I20260806 04:39:24.348909 761 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:24.349021 761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:24.349253 761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:24.351435 761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "d94d179bf56e4374aefde9361954d1d5"
format_stamp: "Formatted at 2026-08-06 04:39:24 on dist-test-slave-vkjp"
server_key: "d92f59d4b68cd88aef80d0ec9a4baa31"
server_key_iv: "ca81a83c6f951dba813f2bf523c2085b"
server_key_version: "encryptionkey@0"
I20260806 04:39:24.352000 761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "d94d179bf56e4374aefde9361954d1d5"
format_stamp: "Formatted at 2026-08-06 04:39:24 on dist-test-slave-vkjp"
server_key: "d92f59d4b68cd88aef80d0ec9a4baa31"
server_key_iv: "ca81a83c6f951dba813f2bf523c2085b"
server_key_version: "encryptionkey@0"
I20260806 04:39:24.356141 761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:39:24.358695 778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:24.360004 761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:24.360140 761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "d94d179bf56e4374aefde9361954d1d5"
format_stamp: "Formatted at 2026-08-06 04:39:24 on dist-test-slave-vkjp"
server_key: "d92f59d4b68cd88aef80d0ec9a4baa31"
server_key_iv: "ca81a83c6f951dba813f2bf523c2085b"
server_key_version: "encryptionkey@0"
I20260806 04:39:24.360252 761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:24.377743 761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:24.380962 761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:24.381191 761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:24.381794 761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:24.382635 761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:24.382709 761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:24.382781 761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:24.382824 761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:24.392459 761 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:38263
I20260806 04:39:24.392488 899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:38263 every 8 connection(s)
I20260806 04:39:24.393343 761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:39:24.396929 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 761
I20260806 04:39:24.397055 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:39:24.397317 32410 external_mini_cluster.cc:1428] Setting key f30573fe9ca6f2a0c5aafac6b061801b
Aug 06 04:39:24 dist-test-slave-vkjp krb5kdc[508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991164, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:39:24.409298 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.395159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:59445 (local address 127.31.166.190:39837)
0806 04:39:24.395315 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:24.395320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:24.396161 (+ 841us) server_negotiation.cc:407] Connection header received
0806 04:39:24.397663 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:24.397668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:24.397727 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:24.397802 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:24.399073 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.399915 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.401860 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.402070 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.402154 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.404950 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:24.404953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:24.404955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:24.404981 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:24.406901 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:24.407519 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:24.407523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:24.407525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:24.407624 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:24.408080 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:24.408083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:24.408085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:24.408292 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:24.408411 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:24.408956 (+ 545us) server_negotiation.cc:299] Negotiation successful
0806 04:39:24.409093 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:39:24.410147 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.395583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39837 (local address 127.31.166.130:59445)
0806 04:39:24.396010 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:24.396054 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:24.397444 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:24.397999 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:24.398006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:24.398354 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:24.398921 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.398936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.400083 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.400086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.401715 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.401731 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.402221 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.402225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.402454 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.403188 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:24.403206 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:24.404758 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:24.407030 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:24.407038 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:24.407057 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:24.407397 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:24.407777 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:24.407780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:24.407783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:24.407949 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:24.408410 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:24.408417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:24.408692 (+ 275us) client_negotiation.cc:769] Sending connection context
0806 04:39:24.408892 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 04:39:24.409251 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
I20260806 04:39:24.411536 900 heartbeater.cc:347] Connected to a master server at 127.31.166.190:39837
I20260806 04:39:24.411829 900 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:24.412516 900 heartbeater.cc:511] Master 127.31.166.190:39837 requested a full tablet report, sending...
I20260806 04:39:24.413802 555 ts_manager.cc:194] Registered new tserver with Master: d94d179bf56e4374aefde9361954d1d5 (127.31.166.130:38263)
I20260806 04:39:24.414469 555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:59445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:39:24.428791 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39837
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:37977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:24.539479 905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:24.539750 905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:24.539840 905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:24.543962 905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:24.544044 905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:24.544127 905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:39:24.549357 905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:37977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39837
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:24.550417 905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:24.551492 905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:24.561291 910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:24.561439 913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:24.561300 911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:24.561733 905 server_base.cc:1034] running on GCE node
I20260806 04:39:24.562132 905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:24.562781 905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:24.564026 905 hybrid_clock.cc:648] HybridClock initialized: now 1785991164564008 us; error 41 us; skew 500 ppm
Aug 06 04:39:24 dist-test-slave-vkjp krb5kdc[508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991164, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:24.566998 905 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:39:24.567955 905 webserver.cc:460] Webserver started at http://127.31.166.131:34535/ using document root <none> and password file <none>
I20260806 04:39:24.568514 905 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:24.568605 905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:24.568848 905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:24.570904 905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda"
format_stamp: "Formatted at 2026-08-06 04:39:24 on dist-test-slave-vkjp"
server_key: "1c2da1eaeea43b68e69c33c464fbfafd"
server_key_iv: "c0c2635fec5fd4f42f1b2981f86440dd"
server_key_version: "encryptionkey@0"
I20260806 04:39:24.571509 905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda"
format_stamp: "Formatted at 2026-08-06 04:39:24 on dist-test-slave-vkjp"
server_key: "1c2da1eaeea43b68e69c33c464fbfafd"
server_key_iv: "c0c2635fec5fd4f42f1b2981f86440dd"
server_key_version: "encryptionkey@0"
I20260806 04:39:24.575277 905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:39:24.577749 920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:24.578958 905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:39:24.579140 905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda"
format_stamp: "Formatted at 2026-08-06 04:39:24 on dist-test-slave-vkjp"
server_key: "1c2da1eaeea43b68e69c33c464fbfafd"
server_key_iv: "c0c2635fec5fd4f42f1b2981f86440dd"
server_key_version: "encryptionkey@0"
I20260806 04:39:24.579264 905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:24.590976 905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:24.594265 905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:24.594483 905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:24.595130 905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:24.595964 905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:24.596036 905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:24.596110 905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:24.596153 905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:24.605654 905 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:37443
I20260806 04:39:24.605683 1045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:37443 every 8 connection(s)
I20260806 04:39:24.606635 905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:39:24.615715 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 905
I20260806 04:39:24.615861 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:39:24.616147 32410 external_mini_cluster.cc:1428] Setting key 36078bc0c48e1142ccb619ee4ed1d0d7
Aug 06 04:39:24 dist-test-slave-vkjp krb5kdc[508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991164, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:24.621212 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.608652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:46909 (local address 127.31.166.190:39837)
0806 04:39:24.608811 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:24.608815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:24.609647 (+ 832us) server_negotiation.cc:407] Connection header received
0806 04:39:24.611044 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:24.611047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:24.611134 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:24.611217 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:24.612419 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.612971 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.614137 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.614370 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.614485 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.617369 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:24.617373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:24.617376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:24.617408 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:24.619122 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:24.619648 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:24.619651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:24.619652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:24.619741 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:24.620131 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:24.620133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:24.620135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:24.620326 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:24.620408 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:24.620950 (+ 542us) server_negotiation.cc:299] Negotiation successful
0806 04:39:24.621085 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 04:39:24.622013 1048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.609026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39837 (local address 127.31.166.131:46909)
0806 04:39:24.609488 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:24.609532 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:24.610821 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:24.611401 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:24.611408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:24.611780 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:24.612256 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.612268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.613108 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.613116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.614004 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.614012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.614532 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.614536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.614704 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.615409 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:24.615426 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:24.617152 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:24.619237 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:24.619243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:24.619255 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:24.619529 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:24.619868 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:24.619871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:24.619872 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:24.620030 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:24.620446 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:24.620452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:24.620692 (+ 240us) client_negotiation.cc:769] Sending connection context
0806 04:39:24.620856 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 04:39:24.621146 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20260806 04:39:24.623189 1046 heartbeater.cc:347] Connected to a master server at 127.31.166.190:39837
I20260806 04:39:24.623414 1046 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:24.623845 1046 heartbeater.cc:511] Master 127.31.166.190:39837 requested a full tablet report, sending...
I20260806 04:39:24.624816 555 ts_manager.cc:194] Registered new tserver with Master: a9ddbfdc1b4f40ef9ede2664df0f1cda (127.31.166.131:37443)
I20260806 04:39:24.625351 555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:46909
I20260806 04:39:24.630677 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:39:24.642405 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.634292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.31.166.190:39837)
0806 04:39:24.634455 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:24.634458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:24.634706 (+ 248us) server_negotiation.cc:407] Connection header received
0806 04:39:24.634817 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:24.634819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:24.634869 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:24.634937 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:24.635673 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.636239 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.637097 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.637313 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.637421 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.638991 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:24.638994 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:24.638996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:24.639025 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:24.640743 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:24.641214 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:24.641216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:24.641217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:24.641292 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:24.641537 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:24.641539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:24.641540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:24.641759 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:24.641859 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:24.642110 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 04:39:24.642206 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:39:24.647701 555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:39:24.650738 555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:39:24.665937 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.659701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:38263 (local address 127.0.0.1:38592)
0806 04:39:24.660602 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:24.660616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:24.660694 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:24.661480 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:24.661486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:24.661507 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:24.661651 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.661665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.662797 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.662802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.664109 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.664117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.665224 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.665228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.665424 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.665473 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 04:39:24.665603 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 04:39:24.665754 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":47,"thread_start_us":118,"threads_started":1}
I20260806 04:39:24.665922 1062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.660687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.31.166.130:38263)
0806 04:39:24.661066 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:24.661081 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:24.661108 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:39:24.661182 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:24.661186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:24.661332 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:24.661488 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:24.661773 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.662676 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.664246 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.665096 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.665214 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.665364 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:24.665645 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 04:39:24.665745 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":147,"threads_started":1}
I20260806 04:39:24.666903 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.660205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:43205 (local address 127.0.0.1:41672)
0806 04:39:24.660923 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:24.660937 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:24.661030 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:24.662352 (+ 1322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:24.662355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:24.662375 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:24.662452 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.662457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.664112 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.664115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.665363 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.665375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.666424 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.666430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.666530 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.666566 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 04:39:24.666668 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 04:39:24.666744 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":4,"thread_start_us":150,"threads_started":1}
I20260806 04:39:24.667382 1061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.660382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.31.166.129:43205)
0806 04:39:24.661891 (+ 1509us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:24.661906 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:24.661929 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:39:24.662008 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:24.662013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:24.662195 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:24.662362 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:24.662587 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.663961 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.665497 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.666295 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.666441 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.666573 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:24.667072 (+ 499us) server_negotiation.cc:299] Negotiation successful
0806 04:39:24.667217 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1397,"thread_start_us":103,"threads_started":1}
I20260806 04:39:24.667413 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.659448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:37443 (local address 127.0.0.1:36216)
0806 04:39:24.660121 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:24.660169 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:24.660311 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:24.661562 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:24.661566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:24.661587 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:24.661711 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.661722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.663440 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.663444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.664702 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.664713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.665811 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.665815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.665909 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.665932 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:39:24.665989 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 04:39:24.666061 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":41,"thread_start_us":93,"threads_started":1}
I20260806 04:39:24.667511 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.659598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36216 (local address 127.31.166.131:37443)
0806 04:39:24.661159 (+ 1561us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:24.661169 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:24.661189 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:39:24.661255 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:24.661259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:24.661415 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:24.661589 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:24.661879 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.663269 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.664829 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.665657 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.666914 (+ 1257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.667021 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:24.667062 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 04:39:24.667358 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1470,"thread_start_us":142,"threads_started":1}
I20260806 04:39:24.669726 688 tablet_service.cc:1467] Processing CreateTablet for tablet a305983d76174eb19186fbf76b1b578e (DEFAULT_TABLE table=test-table [id=7250deb3448c400b95969a6b6eb1cdac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:24.669775 829 tablet_service.cc:1467] Processing CreateTablet for tablet a305983d76174eb19186fbf76b1b578e (DEFAULT_TABLE table=test-table [id=7250deb3448c400b95969a6b6eb1cdac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:24.670102 974 tablet_service.cc:1467] Processing CreateTablet for tablet a305983d76174eb19186fbf76b1b578e (DEFAULT_TABLE table=test-table [id=7250deb3448c400b95969a6b6eb1cdac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:24.670904 688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a305983d76174eb19186fbf76b1b578e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:24.670902 829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a305983d76174eb19186fbf76b1b578e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:24.671000 974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a305983d76174eb19186fbf76b1b578e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:24.677116 1063 tablet_bootstrap.cc:492] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890: Bootstrap starting.
I20260806 04:39:24.678429 1064 tablet_bootstrap.cc:492] T a305983d76174eb19186fbf76b1b578e P d94d179bf56e4374aefde9361954d1d5: Bootstrap starting.
I20260806 04:39:24.679422 1065 tablet_bootstrap.cc:492] T a305983d76174eb19186fbf76b1b578e P a9ddbfdc1b4f40ef9ede2664df0f1cda: Bootstrap starting.
I20260806 04:39:24.679641 1063 tablet_bootstrap.cc:654] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:24.681342 1064 tablet_bootstrap.cc:654] T a305983d76174eb19186fbf76b1b578e P d94d179bf56e4374aefde9361954d1d5: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:24.681509 1065 tablet_bootstrap.cc:654] T a305983d76174eb19186fbf76b1b578e P a9ddbfdc1b4f40ef9ede2664df0f1cda: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:24.681533 1063 log.cc:824] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:24.682140 1064 log.cc:824] T a305983d76174eb19186fbf76b1b578e P d94d179bf56e4374aefde9361954d1d5: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:24.682938 1065 log.cc:824] T a305983d76174eb19186fbf76b1b578e P a9ddbfdc1b4f40ef9ede2664df0f1cda: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:24.683797 1063 tablet_bootstrap.cc:492] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890: No bootstrap required, opened a new log
I20260806 04:39:24.683996 1063 ts_tablet_manager.cc:1397] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20260806 04:39:24.684350 1064 tablet_bootstrap.cc:492] T a305983d76174eb19186fbf76b1b578e P d94d179bf56e4374aefde9361954d1d5: No bootstrap required, opened a new log
I20260806 04:39:24.684517 1064 ts_tablet_manager.cc:1397] T a305983d76174eb19186fbf76b1b578e P d94d179bf56e4374aefde9361954d1d5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:39:24.684927 1065 tablet_bootstrap.cc:492] T a305983d76174eb19186fbf76b1b578e P a9ddbfdc1b4f40ef9ede2664df0f1cda: No bootstrap required, opened a new log
I20260806 04:39:24.685124 1065 ts_tablet_manager.cc:1397] T a305983d76174eb19186fbf76b1b578e P a9ddbfdc1b4f40ef9ede2664df0f1cda: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:39:24.688000 1063 raft_consensus.cc:348] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37443 } } peers { permanent_uuid: "d94d179bf56e4374aefde9361954d1d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38263 } } peers { permanent_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43205 } }
I20260806 04:39:24.688272 1063 raft_consensus.cc:374] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:24.688329 1063 raft_consensus.cc:729] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8ff74281ab4e9eb7ffcc02b65ee890, State: Initialized, Role: FOLLOWER
I20260806 04:39:24.688470 1064 raft_consensus.cc:348] T a305983d76174eb19186fbf76b1b578e P d94d179bf56e4374aefde9361954d1d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37443 } } peers { permanent_uuid: "d94d179bf56e4374aefde9361954d1d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38263 } } peers { permanent_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43205 } }
I20260806 04:39:24.688710 1064 raft_consensus.cc:374] T a305983d76174eb19186fbf76b1b578e P d94d179bf56e4374aefde9361954d1d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:24.688768 1064 raft_consensus.cc:729] T a305983d76174eb19186fbf76b1b578e P d94d179bf56e4374aefde9361954d1d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d94d179bf56e4374aefde9361954d1d5, State: Initialized, Role: FOLLOWER
I20260806 04:39:24.688546 1065 raft_consensus.cc:348] T a305983d76174eb19186fbf76b1b578e P a9ddbfdc1b4f40ef9ede2664df0f1cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37443 } } peers { permanent_uuid: "d94d179bf56e4374aefde9361954d1d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38263 } } peers { permanent_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43205 } }
I20260806 04:39:24.688858 1065 raft_consensus.cc:374] T a305983d76174eb19186fbf76b1b578e P a9ddbfdc1b4f40ef9ede2664df0f1cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:24.688768 1063 consensus_queue.cc:260] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37443 } } peers { permanent_uuid: "d94d179bf56e4374aefde9361954d1d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38263 } } peers { permanent_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43205 } }
I20260806 04:39:24.688923 1065 raft_consensus.cc:729] T a305983d76174eb19186fbf76b1b578e P a9ddbfdc1b4f40ef9ede2664df0f1cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ddbfdc1b4f40ef9ede2664df0f1cda, State: Initialized, Role: FOLLOWER
I20260806 04:39:24.689183 1064 consensus_queue.cc:260] T a305983d76174eb19186fbf76b1b578e P d94d179bf56e4374aefde9361954d1d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37443 } } peers { permanent_uuid: "d94d179bf56e4374aefde9361954d1d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38263 } } peers { permanent_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43205 } }
I20260806 04:39:24.689460 1065 consensus_queue.cc:260] T a305983d76174eb19186fbf76b1b578e P a9ddbfdc1b4f40ef9ede2664df0f1cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37443 } } peers { permanent_uuid: "d94d179bf56e4374aefde9361954d1d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38263 } } peers { permanent_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43205 } }
I20260806 04:39:24.689832 1063 ts_tablet_manager.cc:1428] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 04:39:24.690129 1064 ts_tablet_manager.cc:1428] T a305983d76174eb19186fbf76b1b578e P d94d179bf56e4374aefde9361954d1d5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:39:24.690178 756 heartbeater.cc:503] Master 127.31.166.190:39837 was elected leader, sending a full tablet report...
I20260806 04:39:24.690531 1065 ts_tablet_manager.cc:1428] T a305983d76174eb19186fbf76b1b578e P a9ddbfdc1b4f40ef9ede2664df0f1cda: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 04:39:24.690672 900 heartbeater.cc:503] Master 127.31.166.190:39837 was elected leader, sending a full tablet report...
I20260806 04:39:24.690886 1046 heartbeater.cc:503] Master 127.31.166.190:39837 was elected leader, sending a full tablet report...
W20260806 04:39:24.857650 1047 tablet.cc:2369] T a305983d76174eb19186fbf76b1b578e P a9ddbfdc1b4f40ef9ede2664df0f1cda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:39:24.882854 1070 raft_consensus.cc:482] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:39:24.883145 1070 raft_consensus.cc:504] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37443 } } peers { permanent_uuid: "d94d179bf56e4374aefde9361954d1d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38263 } } peers { permanent_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43205 } }
I20260806 04:39:24.884671 1070 leader_election.cc:288] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9ddbfdc1b4f40ef9ede2664df0f1cda (127.31.166.131:37443), d94d179bf56e4374aefde9361954d1d5 (127.31.166.130:38263)
I20260806 04:39:24.889196 1058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.885112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:60977 (local address 127.31.166.131:37443)
0806 04:39:24.885466 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:24.885471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:24.885540 (+ 69us) server_negotiation.cc:407] Connection header received
0806 04:39:24.885600 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:24.885604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:24.885657 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:24.885734 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:24.886166 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.887024 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.888178 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.888598 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.888738 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.888792 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:24.888991 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 04:39:24.889064 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254}
I20260806 04:39:24.889222 1074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.884928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:37443 (local address 127.31.166.129:60977)
0806 04:39:24.885295 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:24.885325 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:24.885448 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:24.885849 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:24.885852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:24.885871 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:24.886028 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.886035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.887216 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.887219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.888049 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.888058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.888746 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.888749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.888856 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.888884 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 04:39:24.888966 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 04:39:24.889054 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":75,"threads_started":1}
I20260806 04:39:24.889968 1062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.885113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:46201 (local address 127.31.166.130:38263)
0806 04:39:24.885302 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:24.885307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:24.885675 (+ 368us) server_negotiation.cc:407] Connection header received
0806 04:39:24.885800 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:24.885805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:24.885864 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:24.885942 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:24.886293 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.887023 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.888704 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.889265 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.889388 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.889440 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:24.889784 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 04:39:24.889860 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 04:39:24.890283 995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a305983d76174eb19186fbf76b1b578e" candidate_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda" is_pre_election: true
I20260806 04:39:24.890609 995 raft_consensus.cc:2454] T a305983d76174eb19186fbf76b1b578e P a9ddbfdc1b4f40ef9ede2664df0f1cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af8ff74281ab4e9eb7ffcc02b65ee890 in term 0.
I20260806 04:39:24.890920 1075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.884928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:38263 (local address 127.31.166.129:46201)
0806 04:39:24.885579 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:24.885597 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:24.885713 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:24.885988 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:24.885992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:24.886026 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:24.886153 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.886160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.887166 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.887170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.888558 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:24.888569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.889507 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.889509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:24.889648 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.889670 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:39:24.889766 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 04:39:24.889842 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":268,"thread_start_us":130,"threads_started":1}
I20260806 04:39:24.891343 640 leader_election.cc:302] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9ddbfdc1b4f40ef9ede2664df0f1cda, af8ff74281ab4e9eb7ffcc02b65ee890; no voters:
I20260806 04:39:24.891575 850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a305983d76174eb19186fbf76b1b578e" candidate_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d94d179bf56e4374aefde9361954d1d5" is_pre_election: true
I20260806 04:39:24.891623 1070 raft_consensus.cc:2781] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:39:24.891714 1070 raft_consensus.cc:482] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:39:24.891762 1070 raft_consensus.cc:3037] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:24.891886 850 raft_consensus.cc:2454] T a305983d76174eb19186fbf76b1b578e P d94d179bf56e4374aefde9361954d1d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af8ff74281ab4e9eb7ffcc02b65ee890 in term 0.
W20260806 04:39:24.894456 901 tablet.cc:2369] T a305983d76174eb19186fbf76b1b578e P d94d179bf56e4374aefde9361954d1d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:39:24.894616 1070 raft_consensus.cc:504] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37443 } } peers { permanent_uuid: "d94d179bf56e4374aefde9361954d1d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38263 } } peers { permanent_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43205 } }
I20260806 04:39:24.894984 1070 leader_election.cc:288] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [CANDIDATE]: Term 1 election: Requested vote from peers a9ddbfdc1b4f40ef9ede2664df0f1cda (127.31.166.131:37443), d94d179bf56e4374aefde9361954d1d5 (127.31.166.130:38263)
I20260806 04:39:24.895377 995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a305983d76174eb19186fbf76b1b578e" candidate_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda"
I20260806 04:39:24.895463 850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a305983d76174eb19186fbf76b1b578e" candidate_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d94d179bf56e4374aefde9361954d1d5"
I20260806 04:39:24.895516 995 raft_consensus.cc:3037] T a305983d76174eb19186fbf76b1b578e P a9ddbfdc1b4f40ef9ede2664df0f1cda [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:24.895608 850 raft_consensus.cc:3037] T a305983d76174eb19186fbf76b1b578e P d94d179bf56e4374aefde9361954d1d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:24.896642 850 raft_consensus.cc:2454] T a305983d76174eb19186fbf76b1b578e P d94d179bf56e4374aefde9361954d1d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af8ff74281ab4e9eb7ffcc02b65ee890 in term 1.
I20260806 04:39:24.896925 995 raft_consensus.cc:2454] T a305983d76174eb19186fbf76b1b578e P a9ddbfdc1b4f40ef9ede2664df0f1cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af8ff74281ab4e9eb7ffcc02b65ee890 in term 1.
I20260806 04:39:24.896956 637 leader_election.cc:302] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af8ff74281ab4e9eb7ffcc02b65ee890, d94d179bf56e4374aefde9361954d1d5; no voters:
I20260806 04:39:24.897149 1070 raft_consensus.cc:2781] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:24.897397 1070 raft_consensus.cc:686] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [term 1 LEADER]: Becoming Leader. State: Replica: af8ff74281ab4e9eb7ffcc02b65ee890, State: Running, Role: LEADER
I20260806 04:39:24.897716 1070 consensus_queue.cc:237] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37443 } } peers { permanent_uuid: "d94d179bf56e4374aefde9361954d1d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38263 } } peers { permanent_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43205 } }
I20260806 04:39:24.900667 555 catalog_manager.cc:5495] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890 reported cstate change: term changed from 0 to 1, leader changed from <none> to af8ff74281ab4e9eb7ffcc02b65ee890 (127.31.166.129). New cstate: current_term: 1 leader_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9ddbfdc1b4f40ef9ede2664df0f1cda" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d94d179bf56e4374aefde9361954d1d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af8ff74281ab4e9eb7ffcc02b65ee890" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43205 } health_report { overall_health: HEALTHY } } }
W20260806 04:39:24.926645 757 tablet.cc:2369] T a305983d76174eb19186fbf76b1b578e P af8ff74281ab4e9eb7ffcc02b65ee890: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 04:39:24 dist-test-slave-vkjp krb5kdc[508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.129@KRBTEST.COM
I20260806 04:39:24.961004 1061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.951690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.31.166.129:43205)
0806 04:39:24.951860 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:24.951865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:24.952032 (+ 167us) server_negotiation.cc:407] Connection header received
0806 04:39:24.952140 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:24.952142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:24.952188 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:24.952259 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:24.953011 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.953596 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.954571 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.954779 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.954865 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.957443 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:24.957456 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:24.957471 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:24.957496 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:24.959072 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:24.959595 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:24.959601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:24.959602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:24.959701 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:24.959993 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:24.959996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:24.959997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:24.960266 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:24.960478 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:24.960656 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 04:39:24.960791 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 04:39:24 dist-test-slave-vkjp krb5kdc[508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991164, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 04:39:24 dist-test-slave-vkjp krb5kdc[508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991164, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.166.129@KRBTEST.COM
I20260806 04:39:24.992411 1061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:24.981436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.31.166.129:43205)
0806 04:39:24.981581 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:24.981586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:24.981836 (+ 250us) server_negotiation.cc:407] Connection header received
0806 04:39:24.982081 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:24.982092 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:24.982166 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:24.982273 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:24.983236 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.983942 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.985177 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:24.985526 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:24.985599 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:24.988611 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:24.988615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:24.988618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:24.988649 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:24.990265 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:24.990752 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:24.990756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:24.990758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:24.990845 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:24.991182 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:24.991187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:24.991188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:24.991422 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:24.991562 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:24.992105 (+ 543us) server_negotiation.cc:299] Negotiation successful
0806 04:39:24.992231 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 04:39:25 dist-test-slave-vkjp krb5kdc[508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 04:39:25 dist-test-slave-vkjp krb5kdc[508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.129@KRBTEST.COM
I20260806 04:39:25.020255 1061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:25.011020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41680 (local address 127.31.166.129:43205)
0806 04:39:25.011223 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:25.011228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:25.011399 (+ 171us) server_negotiation.cc:407] Connection header received
0806 04:39:25.011583 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:25.011587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:25.011641 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:25.011728 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:25.012462 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:25.012983 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:25.013845 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:25.014072 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:25.014178 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:25.016722 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:25.016726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:25.016728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:25.016753 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:25.018493 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:25.018915 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:25.018918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:25.018920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:25.019002 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:25.019418 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:25.019421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:25.019422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:25.019628 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:25.019767 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:25.020016 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 04:39:25.020118 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260806 04:39:25.021760 666 tablet_service.cc:2908] Rejecting scan request for tablet a305983d76174eb19186fbf76b1b578e: Uninitialized: clean time has not yet been initialized
I20260806 04:39:25.024154 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 621
I20260806 04:39:25.031188 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 761
I20260806 04:39:25.037009 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 905
I20260806 04:39:25.046047 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 522
2026-08-06T04:39:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2551 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:39:25 dist-test-slave-vkjp krb5kdc[1091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:39:25 dist-test-slave-vkjp krb5kdc[1091](info): set up 2 sockets
Aug 06 04:39:25 dist-test-slave-vkjp krb5kdc[1091](info): commencing operation
krb5kdc: starting...
W20260806 04:39:26.128499 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:39:26 dist-test-slave-vkjp krb5kdc[1091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:39:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:39:26.254218 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:33615
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:33615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:26.365104 1105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:26.365406 1105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:26.365448 1105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:26.369577 1105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:26.369637 1105 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 04:39:26.369655 1105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:26.369673 1105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:26.369690 1105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:26.369705 1105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:39:26.375034 1105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:33615
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:33615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:26.376075 1105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:26.377017 1105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:26.385536 1113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:26.385538 1111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:26.385589 1115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:26.385783 1105 server_base.cc:1034] running on GCE node
I20260806 04:39:26.386373 1105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:26.387630 1105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:26.388827 1105 hybrid_clock.cc:648] HybridClock initialized: now 1785991166388812 us; error 26 us; skew 500 ppm
Aug 06 04:39:26 dist-test-slave-vkjp krb5kdc[1091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:26.391719 1105 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:39:26.392551 1105 webserver.cc:460] Webserver started at http://127.31.166.190:43251/ using document root <none> and password file <none>
I20260806 04:39:26.393098 1105 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:26.393158 1105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:26.393329 1105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:26.395426 1105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "bb0fb22b06424059be33dfa84ba539a5"
format_stamp: "Formatted at 2026-08-06 04:39:26 on dist-test-slave-vkjp"
server_key: "2d4c3e8908ff0be41e8e26695b1dbfa3"
server_key_iv: "a838c59cf5722c77befdae68ed7e3505"
server_key_version: "encryptionkey@0"
I20260806 04:39:26.395936 1105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "bb0fb22b06424059be33dfa84ba539a5"
format_stamp: "Formatted at 2026-08-06 04:39:26 on dist-test-slave-vkjp"
server_key: "2d4c3e8908ff0be41e8e26695b1dbfa3"
server_key_iv: "a838c59cf5722c77befdae68ed7e3505"
server_key_version: "encryptionkey@0"
I20260806 04:39:26.399551 1105 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:39:26.402115 1122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:26.403406 1105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:39:26.403541 1105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "bb0fb22b06424059be33dfa84ba539a5"
format_stamp: "Formatted at 2026-08-06 04:39:26 on dist-test-slave-vkjp"
server_key: "2d4c3e8908ff0be41e8e26695b1dbfa3"
server_key_iv: "a838c59cf5722c77befdae68ed7e3505"
server_key_version: "encryptionkey@0"
I20260806 04:39:26.403642 1105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:26.425608 1105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:26.428704 1105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:26.428911 1105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:26.436806 1105 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:33615
I20260806 04:39:26.436843 1186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:33615 every 8 connection(s)
I20260806 04:39:26.437732 1105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:26.440433 1187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:26.440672 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 1105
I20260806 04:39:26.440785 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:26.441072 32410 external_mini_cluster.cc:1428] Setting key 076614a322d521ce34a40c4371379589
I20260806 04:39:26.447257 1187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5: Bootstrap starting.
I20260806 04:39:26.449819 1187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5: Neither blocks nor log segments found. Creating new log.
Aug 06 04:39:26 dist-test-slave-vkjp krb5kdc[1091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:26.450600 1187 log.cc:824] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:26.452677 1187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5: No bootstrap required, opened a new log
I20260806 04:39:26.455520 1187 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0fb22b06424059be33dfa84ba539a5" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33615 } }
I20260806 04:39:26.455766 1187 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:26.455832 1187 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb0fb22b06424059be33dfa84ba539a5, State: Initialized, Role: FOLLOWER
I20260806 04:39:26.456283 1187 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0fb22b06424059be33dfa84ba539a5" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33615 } }
I20260806 04:39:26.456452 1187 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:26.456533 1187 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:26.456656 1187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:26.456964 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:26.442425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.31.166.190:33615)
0806 04:39:26.442824 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:26.442852 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:26.442893 (+ 41us) server_negotiation.cc:407] Connection header received
0806 04:39:26.443680 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:26.443703 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:26.444111 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:26.444453 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:26.445108 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.446577 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.447536 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.448207 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.448381 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:26.451145 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:26.451152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:26.451175 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:26.451205 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:26.453716 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:26.454215 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:26.454226 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:26.454231 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:26.454359 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:26.454688 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:26.454698 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:26.454700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:26.455178 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:26.455412 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:26.455736 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 04:39:26.456072 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
I20260806 04:39:26.457605 1187 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0fb22b06424059be33dfa84ba539a5" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33615 } }
I20260806 04:39:26.457935 1187 leader_election.cc:302] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb0fb22b06424059be33dfa84ba539a5; no voters:
I20260806 04:39:26.458259 1187 leader_election.cc:288] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:26.458354 1193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:26.458529 1193 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [term 1 LEADER]: Becoming Leader. State: Replica: bb0fb22b06424059be33dfa84ba539a5, State: Running, Role: LEADER
I20260806 04:39:26.458807 1193 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0fb22b06424059be33dfa84ba539a5" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33615 } }
I20260806 04:39:26.459528 1187 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:26.460688 1195 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb0fb22b06424059be33dfa84ba539a5. Latest consensus state: current_term: 1 leader_uuid: "bb0fb22b06424059be33dfa84ba539a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0fb22b06424059be33dfa84ba539a5" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33615 } } }
I20260806 04:39:26.460698 1194 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb0fb22b06424059be33dfa84ba539a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb0fb22b06424059be33dfa84ba539a5" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33615 } } }
I20260806 04:39:26.460816 1195 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:26.460847 1194 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:26.462100 1202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:26.465142 1202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:26.470487 1202 catalog_manager.cc:1347] Generated new cluster ID: 1a05bd4fb5174c62bceba085737a12cd
I20260806 04:39:26.470557 1202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:26.508677 1202 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:26.509699 1202 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:26.524013 1202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5: Generated new TSK 0
I20260806 04:39:26.524730 1202 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:39:26.558140 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33615
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:26.669364 1217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:26.669665 1217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:26.669711 1217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:26.673893 1217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:26.673949 1217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:26.673987 1217 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:39:26.674021 1217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:39:26.679164 1217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.166.190:33615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:26.680183 1217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:26.681216 1217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:26.690959 1222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:26.690997 1217 server_base.cc:1034] running on GCE node
W20260806 04:39:26.690959 1223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:26.690961 1225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:26.691668 1217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:26.692337 1217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:26.693538 1217 hybrid_clock.cc:648] HybridClock initialized: now 1785991166693524 us; error 26 us; skew 500 ppm
Aug 06 04:39:26 dist-test-slave-vkjp krb5kdc[1091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:26.696516 1217 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:39:26.697476 1217 webserver.cc:460] Webserver started at http://127.31.166.129:36709/ using document root <none> and password file <none>
I20260806 04:39:26.698024 1217 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:26.698089 1217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:26.698309 1217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:26.700646 1217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "a63bc75001934e98a9c827ee83d10107"
format_stamp: "Formatted at 2026-08-06 04:39:26 on dist-test-slave-vkjp"
server_key: "0180dd9348cc3347b147d1ff82ae8aa3"
server_key_iv: "6540fcde3b87457d388e31c6a2980ccc"
server_key_version: "encryptionkey@0"
I20260806 04:39:26.701201 1217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "a63bc75001934e98a9c827ee83d10107"
format_stamp: "Formatted at 2026-08-06 04:39:26 on dist-test-slave-vkjp"
server_key: "0180dd9348cc3347b147d1ff82ae8aa3"
server_key_iv: "6540fcde3b87457d388e31c6a2980ccc"
server_key_version: "encryptionkey@0"
I20260806 04:39:26.705255 1217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:39:26.707823 1233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:26.709178 1217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:39:26.709285 1217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "a63bc75001934e98a9c827ee83d10107"
format_stamp: "Formatted at 2026-08-06 04:39:26 on dist-test-slave-vkjp"
server_key: "0180dd9348cc3347b147d1ff82ae8aa3"
server_key_iv: "6540fcde3b87457d388e31c6a2980ccc"
server_key_version: "encryptionkey@0"
I20260806 04:39:26.709439 1217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:26.721467 1217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:26.724867 1217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:26.725080 1217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:26.726522 1217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:26.726600 1217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:26.726680 1217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:26.726727 1217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:39:26 dist-test-slave-vkjp krb5kdc[1091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:26.738231 1217 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:39125
I20260806 04:39:26.738252 1358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:39125 every 8 connection(s)
I20260806 04:39:26.739240 1217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:39:26.740695 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:26.727500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:34849 (local address 127.31.166.190:33615)
0806 04:39:26.727634 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:26.727638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:26.728698 (+ 1060us) server_negotiation.cc:407] Connection header received
0806 04:39:26.729704 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:26.729708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:26.729773 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:26.729837 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:26.731276 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.732222 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.733605 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.733899 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.734028 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:26.736992 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:26.736996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:26.736999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:26.737035 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:26.738579 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:26.739116 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:26.739123 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:26.739127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:26.739214 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:26.739556 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:26.739563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:26.739565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:26.739822 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:26.739921 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:26.740463 (+ 542us) server_negotiation.cc:299] Negotiation successful
0806 04:39:26.740575 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:39:26.741477 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:26.728025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33615 (local address 127.31.166.129:34849)
0806 04:39:26.728539 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:26.728580 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:26.729445 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:26.730252 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:26.730258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:26.730592 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:26.731062 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:26.731075 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.732420 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.732424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:26.733462 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:26.733472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.734077 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.734081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:26.734266 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:26.735015 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:26.735043 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:26.736798 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:26.738701 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:26.738706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:26.738720 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:26.738987 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:26.739308 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:26.739312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:26.739314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:26.739459 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:26.739985 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:26.739991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:26.740227 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 04:39:26.740389 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 04:39:26.740630 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":129,"threads_started":1}
I20260806 04:39:26.742739 1359 heartbeater.cc:347] Connected to a master server at 127.31.166.190:33615
I20260806 04:39:26.742951 1359 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:26.743563 1359 heartbeater.cc:511] Master 127.31.166.190:33615 requested a full tablet report, sending...
I20260806 04:39:26.744655 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 1217
I20260806 04:39:26.744772 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:39:26.745101 32410 external_mini_cluster.cc:1428] Setting key 2baaf7b962e6196d9b6dfbd5a884a089
I20260806 04:39:26.745215 1139 ts_manager.cc:194] Registered new tserver with Master: a63bc75001934e98a9c827ee83d10107 (127.31.166.129:39125)
I20260806 04:39:26.746694 1139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:34849
I20260806 04:39:26.752671 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:26.745296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:36663 (local address 127.31.166.190:33615)
0806 04:39:26.745417 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:26.745421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:26.745685 (+ 264us) server_negotiation.cc:407] Connection header received
0806 04:39:26.745782 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:26.745784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:26.745823 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:26.745906 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:26.746406 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.746972 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.747974 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.748290 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.748432 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:26.749548 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:26.749552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:26.749554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:26.749578 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:26.751012 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:26.751466 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:26.751469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:26.751471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:26.751553 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:26.751828 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:26.751832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:26.751834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:26.752013 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:26.752121 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:26.752374 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 04:39:26.752481 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:39:26.752683 1365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:26.745277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33615 (local address 127.31.166.129:36663)
0806 04:39:26.745592 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:26.745607 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:26.745703 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:26.745954 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:26.745956 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:26.746166 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:26.746284 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:26.746290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.747130 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.747134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:26.747846 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:26.747856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.748436 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.748439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:26.748531 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:26.749072 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:26.749083 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:26.749411 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:26.751148 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:26.751153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:26.751155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:26.751363 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:26.751644 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:26.751648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:26.751650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:26.751737 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:26.752111 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:26.752115 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:26.752211 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 04:39:26.752318 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 04:39:26.752468 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:39:26.775399 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33615
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:26.887611 1368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:26.887872 1368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:26.887917 1368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:26.892068 1368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:26.892122 1368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:26.892187 1368 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:39:26.892274 1368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:39:26.897480 1368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.166.190:33615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:26.898504 1368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:26.899551 1368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:26.909145 1374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:26.909145 1373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:26.909171 1376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:26.909235 1368 server_base.cc:1034] running on GCE node
I20260806 04:39:26.909754 1368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:26.910406 1368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:26.911617 1368 hybrid_clock.cc:648] HybridClock initialized: now 1785991166911576 us; error 62 us; skew 500 ppm
Aug 06 04:39:26 dist-test-slave-vkjp krb5kdc[1091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:26.914929 1368 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:39:26.915932 1368 webserver.cc:460] Webserver started at http://127.31.166.130:41653/ using document root <none> and password file <none>
I20260806 04:39:26.916515 1368 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:26.916611 1368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:26.916848 1368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:26.918908 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "cbd396205d3147fca269a2c4f8b0ed16"
format_stamp: "Formatted at 2026-08-06 04:39:26 on dist-test-slave-vkjp"
server_key: "ea36ad298116754091a6171ab72af28a"
server_key_iv: "dce239022d4cd542a2f9093f4951457d"
server_key_version: "encryptionkey@0"
I20260806 04:39:26.919559 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "cbd396205d3147fca269a2c4f8b0ed16"
format_stamp: "Formatted at 2026-08-06 04:39:26 on dist-test-slave-vkjp"
server_key: "ea36ad298116754091a6171ab72af28a"
server_key_iv: "dce239022d4cd542a2f9093f4951457d"
server_key_version: "encryptionkey@0"
I20260806 04:39:26.923455 1368 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:39:26.925957 1383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:26.927253 1368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:26.927482 1368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "cbd396205d3147fca269a2c4f8b0ed16"
format_stamp: "Formatted at 2026-08-06 04:39:26 on dist-test-slave-vkjp"
server_key: "ea36ad298116754091a6171ab72af28a"
server_key_iv: "dce239022d4cd542a2f9093f4951457d"
server_key_version: "encryptionkey@0"
I20260806 04:39:26.927685 1368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:26.949991 1368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:26.953681 1368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:26.953933 1368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:26.955541 1368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:26.955616 1368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:26.955710 1368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:26.955751 1368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:39:26 dist-test-slave-vkjp krb5kdc[1091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:26.967374 1368 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:41115
I20260806 04:39:26.967432 1505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:41115 every 8 connection(s)
I20260806 04:39:26.968505 1368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:39:26.971596 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:26.956338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:58555 (local address 127.31.166.190:33615)
0806 04:39:26.956506 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:26.956512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:26.957944 (+ 1432us) server_negotiation.cc:407] Connection header received
0806 04:39:26.959325 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:26.959329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:26.959393 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:26.959498 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:26.961107 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.962143 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.963819 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.964076 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.964203 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:26.967332 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:26.967336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:26.967339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:26.967375 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:26.969220 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:26.969856 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:26.969860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:26.969862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:26.969950 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:26.970384 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:26.970387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:26.970389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:26.970587 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:26.970674 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:26.971251 (+ 577us) server_negotiation.cc:299] Negotiation successful
0806 04:39:26.971429 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 04:39:26.972232 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 1368
I20260806 04:39:26.972359 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:39:26.972312 1395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:26.957228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33615 (local address 127.31.166.130:58555)
0806 04:39:26.957789 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:26.957839 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:26.959040 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:26.959693 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:26.959705 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:26.960228 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:26.960921 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:26.960944 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.962428 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.962434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:26.963681 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:26.963692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.964274 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.964280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:26.964466 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:26.965183 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:26.965207 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:26.967082 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:26.969371 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:26.969377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:26.969389 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:26.969718 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:26.970045 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:26.970049 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:26.970051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:26.970207 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:26.970717 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:26.970723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:26.970967 (+ 244us) client_negotiation.cc:769] Sending connection context
0806 04:39:26.971193 (+ 226us) client_negotiation.cc:240] Negotiation successful
0806 04:39:26.971519 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":139,"threads_started":1}
I20260806 04:39:26.972680 32410 external_mini_cluster.cc:1428] Setting key c01c8703ab3c5f6abb8c3d309d00d8a0
I20260806 04:39:26.973837 1506 heartbeater.cc:347] Connected to a master server at 127.31.166.190:33615
I20260806 04:39:26.973973 1506 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:26.974813 1506 heartbeater.cc:511] Master 127.31.166.190:33615 requested a full tablet report, sending...
I20260806 04:39:26.975950 1138 ts_manager.cc:194] Registered new tserver with Master: cbd396205d3147fca269a2c4f8b0ed16 (127.31.166.130:41115)
I20260806 04:39:26.976706 1138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:58555
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:39:26.985110 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:26.976206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:39923 (local address 127.31.166.190:33615)
0806 04:39:26.976306 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:26.976310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:26.976629 (+ 319us) server_negotiation.cc:407] Connection header received
0806 04:39:26.976801 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:26.976804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:26.976844 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:26.976894 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:26.977506 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.978195 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.979535 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.979790 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.979897 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:26.981379 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:26.981382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:26.981384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:26.981409 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:26.983376 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:26.983869 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:26.983872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:26.983874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:26.983969 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:26.984246 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:26.984249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:26.984250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:26.984453 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:26.984531 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:26.984904 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 04:39:26.985001 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260806 04:39:26.985109 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:26.976155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33615 (local address 127.31.166.130:39923)
0806 04:39:26.976550 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:26.976565 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:26.976694 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:26.977034 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:26.977036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:26.977265 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:26.977389 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:26.977395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.978348 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.978355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:26.979386 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:26.979399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:26.979931 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:26.979938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:26.980075 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:26.980654 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:26.980674 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:26.981186 (+ 512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:26.983500 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:26.983503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:26.983505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:26.983710 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:26.984069 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:26.984071 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:26.984073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:26.984158 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:26.984541 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:26.984544 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:26.984702 (+ 158us) client_negotiation.cc:769] Sending connection context
0806 04:39:26.984840 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 04:39:26.984963 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":205,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:39:27.004326 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33615
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:37767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:27.117012 1517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:27.117276 1517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:27.117348 1517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:27.121726 1517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:27.121784 1517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:27.121832 1517 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:39:27.121917 1517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:39:27.127225 1517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:37767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.166.190:33615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:27.128279 1517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:27.129320 1517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:27.139077 1525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:27.139138 1522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:27.139214 1517 server_base.cc:1034] running on GCE node
W20260806 04:39:27.139138 1523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:27.139883 1517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:27.140487 1517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:27.141672 1517 hybrid_clock.cc:648] HybridClock initialized: now 1785991167141640 us; error 45 us; skew 500 ppm
Aug 06 04:39:27 dist-test-slave-vkjp krb5kdc[1091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:27.144796 1517 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:39:27.145701 1517 webserver.cc:460] Webserver started at http://127.31.166.131:42775/ using document root <none> and password file <none>
I20260806 04:39:27.146306 1517 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:27.146374 1517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:27.146606 1517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:27.148855 1517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21"
format_stamp: "Formatted at 2026-08-06 04:39:27 on dist-test-slave-vkjp"
server_key: "2cd45d15e02b5fe7824e16cbd0bef66f"
server_key_iv: "7f891a24230b63acdb022ba769da64d7"
server_key_version: "encryptionkey@0"
I20260806 04:39:27.149503 1517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21"
format_stamp: "Formatted at 2026-08-06 04:39:27 on dist-test-slave-vkjp"
server_key: "2cd45d15e02b5fe7824e16cbd0bef66f"
server_key_iv: "7f891a24230b63acdb022ba769da64d7"
server_key_version: "encryptionkey@0"
I20260806 04:39:27.153314 1517 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260806 04:39:27.155823 1533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:27.156987 1517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:39:27.157112 1517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21"
format_stamp: "Formatted at 2026-08-06 04:39:27 on dist-test-slave-vkjp"
server_key: "2cd45d15e02b5fe7824e16cbd0bef66f"
server_key_iv: "7f891a24230b63acdb022ba769da64d7"
server_key_version: "encryptionkey@0"
I20260806 04:39:27.157219 1517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:27.172111 1517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:27.175486 1517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:27.175686 1517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:27.177084 1517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:27.177134 1517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:27.177196 1517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:27.177230 1517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:39:27 dist-test-slave-vkjp krb5kdc[1091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991167, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:27.187958 1517 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:44573
I20260806 04:39:27.187988 1655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:44573 every 8 connection(s)
I20260806 04:39:27.189013 1517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:39:27.191301 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 1517
I20260806 04:39:27.191422 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:39:27.191449 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.178047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:58233 (local address 127.31.166.190:33615)
0806 04:39:27.178209 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:27.178214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:27.179030 (+ 816us) server_negotiation.cc:407] Connection header received
0806 04:39:27.180227 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:27.180233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:27.180302 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:27.180417 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:27.181937 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.182817 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.184188 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.184669 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.184791 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.187396 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:27.187402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:27.187408 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:27.187443 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:27.189232 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:27.189851 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:27.189855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:27.189856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:27.189953 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:27.190264 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:27.190266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:27.190268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:27.190528 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:27.190610 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:27.191055 (+ 445us) server_negotiation.cc:299] Negotiation successful
0806 04:39:27.191224 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 04:39:27.191741 32410 external_mini_cluster.cc:1428] Setting key 06fe773fca0175cda8643ce1fa94dc45
I20260806 04:39:27.192201 1542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.178421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33615 (local address 127.31.166.131:58233)
0806 04:39:27.178854 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:27.178903 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:27.179966 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:27.180635 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:27.180645 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:27.181055 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:27.181762 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.181785 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.182944 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.182949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.184051 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.184061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.184811 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.184814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.184953 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.185521 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:27.185536 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:27.187222 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:27.189394 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:27.189400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:27.189414 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:27.189725 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:27.190048 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:27.190051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:27.190052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:27.190176 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:27.190605 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:27.190609 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:27.190807 (+ 198us) client_negotiation.cc:769] Sending connection context
0806 04:39:27.190950 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 04:39:27.191240 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":128,"threads_started":1}
I20260806 04:39:27.193483 1656 heartbeater.cc:347] Connected to a master server at 127.31.166.190:33615
I20260806 04:39:27.193629 1656 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:27.194224 1656 heartbeater.cc:511] Master 127.31.166.190:33615 requested a full tablet report, sending...
I20260806 04:39:27.195259 1139 ts_manager.cc:194] Registered new tserver with Master: 2aa3f2eb048f4fe2b5344cc39ec27a21 (127.31.166.131:44573)
I20260806 04:39:27.195997 1139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:58233
I20260806 04:39:27.202765 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.195670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:56779 (local address 127.31.166.190:33615)
0806 04:39:27.195786 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:27.195789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:27.196458 (+ 669us) server_negotiation.cc:407] Connection header received
0806 04:39:27.196566 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:27.196568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:27.196606 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:27.196680 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:27.197183 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.197692 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.198534 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.198751 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.198831 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.199994 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:27.199997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:27.199999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:27.200020 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:27.201473 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:27.201827 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:27.201830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:27.201831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:27.201907 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:27.202129 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:27.202130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:27.202132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:27.202284 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:27.202390 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:27.202573 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 04:39:27.202656 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:39:27.202777 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.195622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33615 (local address 127.31.166.131:56779)
0806 04:39:27.196362 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:27.196376 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:27.196462 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:27.196719 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:27.196722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:27.196962 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:27.197070 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.197077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.197831 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.197836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.198423 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.198432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.198873 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.198876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.198944 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.199542 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:27.199553 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:27.199873 (+ 320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:27.201564 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:27.201566 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:27.201567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:27.201749 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:27.201984 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:27.201986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:27.201987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:27.202056 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:27.202384 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:27.202385 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:27.202457 (+ 72us) client_negotiation.cc:769] Sending connection context
0806 04:39:27.202535 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 04:39:27.202636 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":87,"threads_started":1}
I20260806 04:39:27.205722 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 04:39:27 dist-test-slave-vkjp krb5kdc[1091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991167, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 04:39:27 dist-test-slave-vkjp krb5kdc[1091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991167, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:27.232245 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.223421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48306 (local address 127.31.166.190:33615)
0806 04:39:27.223565 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:27.223569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:27.223582 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:39:27.223626 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:27.223628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:27.223668 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:27.223749 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:27.224520 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.225069 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.225863 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.226070 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.226146 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.228861 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:27.228865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:27.228867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:27.228897 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:27.230344 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:27.230886 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:27.230892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:27.230894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:27.230993 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:27.231318 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:27.231322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:27.231324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:27.231528 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:27.231673 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:27.231928 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 04:39:27.232075 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:39:27.235241 1139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:39:27.238137 1139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:39:27.252792 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.247128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.31.166.129:39125)
0806 04:39:27.248144 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:27.248154 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:27.248180 (+ 26us) server_negotiation.cc:407] Connection header received
0806 04:39:27.248264 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:27.248272 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:27.248508 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:27.248669 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:27.249001 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.250026 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.251538 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.252081 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.252215 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.252351 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:27.252554 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 04:39:27.252647 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":905,"thread_start_us":92,"threads_started":1}
I20260806 04:39:27.252784 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.246911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:39125 (local address 127.0.0.1:35422)
0806 04:39:27.247635 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:27.247695 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:27.247846 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:27.248675 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:27.248681 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:27.248705 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:27.248850 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.248861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.250138 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.250143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.251409 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.251418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.252214 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.252218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.252327 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.252367 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 04:39:27.252516 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 04:39:27.252635 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"spinlock_wait_cycles":6016,"thread_start_us":86,"threads_started":1}
I20260806 04:39:27.253383 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.247237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:41115 (local address 127.0.0.1:46974)
0806 04:39:27.247635 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:27.247695 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:27.247843 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:27.248925 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:27.248928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:27.248949 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:27.249075 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.249090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.250683 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.250687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.251718 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.251728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.252938 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.252943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.253052 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.253076 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 04:39:27.253125 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 04:39:27.253191 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":20,"spinlock_wait_cycles":4864,"thread_start_us":109,"threads_started":1}
I20260806 04:39:27.254350 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.247449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.31.166.130:41115)
0806 04:39:27.248458 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:27.248478 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:27.248502 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:39:27.248579 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:27.248587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:27.248760 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:27.248945 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:27.249204 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.250544 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.251854 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.252738 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.253905 (+ 1167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.254036 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:27.254088 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 04:39:27.254175 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":131,"threads_started":1}
I20260806 04:39:27.254745 1677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.247656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:44573 (local address 127.0.0.1:60200)
0806 04:39:27.248099 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:27.248113 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:27.248196 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:27.249068 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:27.249071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:27.249088 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:27.249164 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.249170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.250683 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.250687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.253211 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.253220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.254369 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.254373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.254458 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.254492 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 04:39:27.254561 (+ 69us) client_negotiation.cc:240] Negotiation successful
0806 04:39:27.254626 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260806 04:39:27.254864 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.248054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60200 (local address 127.31.166.131:44573)
0806 04:39:27.248480 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:27.248489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:27.248508 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:39:27.248570 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:27.248573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:27.248773 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:27.248928 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:27.249273 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.250530 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.253352 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.254239 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.254399 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.254547 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:27.254600 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 04:39:27.254703 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":122,"threads_started":1}
I20260806 04:39:27.256124 1287 tablet_service.cc:1467] Processing CreateTablet for tablet cef7ceac37e440fdb7d2078a95e72b7a (DEFAULT_TABLE table=test-table [id=a94b32e495bd42b788a99363b25d1f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:27.257072 1287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef7ceac37e440fdb7d2078a95e72b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:27.257323 1422 tablet_service.cc:1467] Processing CreateTablet for tablet cef7ceac37e440fdb7d2078a95e72b7a (DEFAULT_TABLE table=test-table [id=a94b32e495bd42b788a99363b25d1f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:27.257539 1585 tablet_service.cc:1467] Processing CreateTablet for tablet cef7ceac37e440fdb7d2078a95e72b7a (DEFAULT_TABLE table=test-table [id=a94b32e495bd42b788a99363b25d1f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:27.258378 1422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef7ceac37e440fdb7d2078a95e72b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:27.258419 1585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef7ceac37e440fdb7d2078a95e72b7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:27.263983 1679 tablet_bootstrap.cc:492] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107: Bootstrap starting.
I20260806 04:39:27.266052 1679 tablet_bootstrap.cc:654] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:27.266851 1679 log.cc:824] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:27.267052 1680 tablet_bootstrap.cc:492] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16: Bootstrap starting.
I20260806 04:39:27.268519 1682 tablet_bootstrap.cc:492] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21: Bootstrap starting.
I20260806 04:39:27.268992 1680 tablet_bootstrap.cc:654] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:27.269641 1680 log.cc:824] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:27.270352 1682 tablet_bootstrap.cc:654] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:27.271260 1682 log.cc:824] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:27.271917 1680 tablet_bootstrap.cc:492] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16: No bootstrap required, opened a new log
I20260806 04:39:27.272156 1680 ts_tablet_manager.cc:1397] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.003s
I20260806 04:39:27.272487 1679 tablet_bootstrap.cc:492] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107: No bootstrap required, opened a new log
I20260806 04:39:27.272710 1679 ts_tablet_manager.cc:1397] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260806 04:39:27.273535 1682 tablet_bootstrap.cc:492] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21: No bootstrap required, opened a new log
I20260806 04:39:27.273722 1682 ts_tablet_manager.cc:1397] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 04:39:27.276185 1679 raft_consensus.cc:348] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } }
I20260806 04:39:27.276430 1679 raft_consensus.cc:374] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:27.276546 1679 raft_consensus.cc:729] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a63bc75001934e98a9c827ee83d10107, State: Initialized, Role: FOLLOWER
I20260806 04:39:27.276614 1680 raft_consensus.cc:348] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } }
I20260806 04:39:27.276889 1680 raft_consensus.cc:374] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:27.276969 1680 raft_consensus.cc:729] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbd396205d3147fca269a2c4f8b0ed16, State: Initialized, Role: FOLLOWER
I20260806 04:39:27.277025 1679 consensus_queue.cc:260] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } }
I20260806 04:39:27.277433 1680 consensus_queue.cc:260] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } }
I20260806 04:39:27.278065 1679 ts_tablet_manager.cc:1428] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 04:39:27.277895 1682 raft_consensus.cc:348] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } }
I20260806 04:39:27.278146 1682 raft_consensus.cc:374] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:27.278275 1682 raft_consensus.cc:729] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa3f2eb048f4fe2b5344cc39ec27a21, State: Initialized, Role: FOLLOWER
I20260806 04:39:27.278505 1359 heartbeater.cc:503] Master 127.31.166.190:33615 was elected leader, sending a full tablet report...
I20260806 04:39:27.278748 1682 consensus_queue.cc:260] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } }
I20260806 04:39:27.279421 1656 heartbeater.cc:503] Master 127.31.166.190:33615 was elected leader, sending a full tablet report...
I20260806 04:39:27.279776 1682 ts_tablet_manager.cc:1428] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260806 04:39:27.279842 1680 ts_tablet_manager.cc:1428] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260806 04:39:27.280196 1506 heartbeater.cc:503] Master 127.31.166.190:33615 was elected leader, sending a full tablet report...
I20260806 04:39:27.406816 1686 raft_consensus.cc:482] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:39:27.407059 1686 raft_consensus.cc:504] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } }
I20260806 04:39:27.408326 1686 leader_election.cc:288] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2aa3f2eb048f4fe2b5344cc39ec27a21 (127.31.166.131:44573), cbd396205d3147fca269a2c4f8b0ed16 (127.31.166.130:41115)
I20260806 04:39:27.412513 1676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.408745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:37257 (local address 127.31.166.130:41115)
0806 04:39:27.408942 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:27.408947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:27.409144 (+ 197us) server_negotiation.cc:407] Connection header received
0806 04:39:27.409297 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:27.409300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:27.409348 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:27.409426 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:27.409754 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.410273 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.411535 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.411938 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.412066 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.412121 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:27.412298 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 04:39:27.412367 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 04:39:27.412544 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.408624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:41115 (local address 127.31.166.129:37257)
0806 04:39:27.409050 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:27.409062 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:27.409189 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:27.409450 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:27.409452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:27.409468 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:27.409643 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.409649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.410380 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.410385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.411416 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.411425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.412059 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.412063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.412162 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.412184 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:39:27.412283 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 04:39:27.412353 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":69,"spinlock_wait_cycles":21888,"thread_start_us":132,"threads_started":1}
I20260806 04:39:27.412568 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.408673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:44573 (local address 127.31.166.129:48813)
0806 04:39:27.409062 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:27.409074 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:27.409208 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:27.409482 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:27.409485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:27.409519 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:27.409681 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.409686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.410412 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.410417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.411426 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.411435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.412097 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.412100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.412213 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.412239 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:39:27.412326 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 04:39:27.412396 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":69,"spinlock_wait_cycles":3200,"thread_start_us":116,"threads_started":1}
I20260806 04:39:27.412547 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.408761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:48813 (local address 127.31.166.131:44573)
0806 04:39:27.408912 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:27.408917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:27.409154 (+ 237us) server_negotiation.cc:407] Connection header received
0806 04:39:27.409329 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:27.409332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:27.409379 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:27.409461 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:27.409791 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.410293 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.411547 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.411945 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.412085 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.412136 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:27.412344 (+ 208us) server_negotiation.cc:299] Negotiation successful
0806 04:39:27.412421 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:39:27.413205 1457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef7ceac37e440fdb7d2078a95e72b7a" candidate_uuid: "a63bc75001934e98a9c827ee83d10107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbd396205d3147fca269a2c4f8b0ed16" is_pre_election: true
I20260806 04:39:27.413314 1608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef7ceac37e440fdb7d2078a95e72b7a" candidate_uuid: "a63bc75001934e98a9c827ee83d10107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" is_pre_election: true
I20260806 04:39:27.413552 1608 raft_consensus.cc:2454] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a63bc75001934e98a9c827ee83d10107 in term 0.
I20260806 04:39:27.413539 1457 raft_consensus.cc:2454] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a63bc75001934e98a9c827ee83d10107 in term 0.
I20260806 04:39:27.414108 1235 leader_election.cc:302] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2aa3f2eb048f4fe2b5344cc39ec27a21, a63bc75001934e98a9c827ee83d10107, cbd396205d3147fca269a2c4f8b0ed16; no voters:
I20260806 04:39:27.414426 1686 raft_consensus.cc:2781] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:39:27.414507 1686 raft_consensus.cc:482] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:39:27.414556 1686 raft_consensus.cc:3037] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:27.415637 1686 raft_consensus.cc:504] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } }
I20260806 04:39:27.416031 1686 leader_election.cc:288] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [CANDIDATE]: Term 1 election: Requested vote from peers 2aa3f2eb048f4fe2b5344cc39ec27a21 (127.31.166.131:44573), cbd396205d3147fca269a2c4f8b0ed16 (127.31.166.130:41115)
I20260806 04:39:27.416370 1608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef7ceac37e440fdb7d2078a95e72b7a" candidate_uuid: "a63bc75001934e98a9c827ee83d10107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21"
I20260806 04:39:27.416417 1457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef7ceac37e440fdb7d2078a95e72b7a" candidate_uuid: "a63bc75001934e98a9c827ee83d10107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbd396205d3147fca269a2c4f8b0ed16"
I20260806 04:39:27.416540 1608 raft_consensus.cc:3037] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:27.416543 1457 raft_consensus.cc:3037] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:27.417614 1457 raft_consensus.cc:2454] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a63bc75001934e98a9c827ee83d10107 in term 1.
I20260806 04:39:27.417963 1237 leader_election.cc:302] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a63bc75001934e98a9c827ee83d10107, cbd396205d3147fca269a2c4f8b0ed16; no voters:
I20260806 04:39:27.417999 1608 raft_consensus.cc:2454] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a63bc75001934e98a9c827ee83d10107 in term 1.
I20260806 04:39:27.418131 1686 raft_consensus.cc:2781] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:27.418392 1686 raft_consensus.cc:686] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [term 1 LEADER]: Becoming Leader. State: Replica: a63bc75001934e98a9c827ee83d10107, State: Running, Role: LEADER
I20260806 04:39:27.418808 1686 consensus_queue.cc:237] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } }
I20260806 04:39:27.422019 1138 catalog_manager.cc:5495] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 reported cstate change: term changed from 0 to 1, leader changed from <none> to a63bc75001934e98a9c827ee83d10107 (127.31.166.129). New cstate: current_term: 1 leader_uuid: "a63bc75001934e98a9c827ee83d10107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } health_report { overall_health: UNKNOWN } } }
W20260806 04:39:27.439976 1657 tablet.cc:2369] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:39:27.456645 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.453030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.31.166.129:39125)
0806 04:39:27.453194 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:27.453198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:27.453213 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:39:27.453387 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:27.453389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:27.453448 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:27.453531 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:27.453772 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.454283 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.455120 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.455332 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.455411 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.455758 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:27.456166 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:27.456263 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:27.456388 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 04:39:27.456446 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 04:39:27.462809 1608 raft_consensus.cc:1261] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 1 FOLLOWER]: Refusing update from remote peer a63bc75001934e98a9c827ee83d10107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:27.462809 1457 raft_consensus.cc:1261] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16 [term 1 FOLLOWER]: Refusing update from remote peer a63bc75001934e98a9c827ee83d10107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:27.463557 1686 consensus_queue.cc:1035] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:27.463790 1691 consensus_queue.cc:1035] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 04:39:27.470781 1508 tablet.cc:2369] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:39:27.471199 1694 mvcc.cc:205] Tried to move back new op lower bound from 7315419821920088064 to 7315419821750415360. Current Snapshot: MvccSnapshot[applied={T|T < 7315419821920088064}]
I20260806 04:39:27.473184 1695 mvcc.cc:205] Tried to move back new op lower bound from 7315419821920088064 to 7315419821750415360. Current Snapshot: MvccSnapshot[applied={T|T < 7315419821920088064}]
I20260806 04:39:27.473618 1696 mvcc.cc:205] Tried to move back new op lower bound from 7315419821920088064 to 7315419821750415360. Current Snapshot: MvccSnapshot[applied={T|T < 7315419821920088064}]
I20260806 04:39:27.483314 1136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:27.483568 1136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:27.486061 1136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5: Sending DeleteTablet for 3 replicas of tablet cef7ceac37e440fdb7d2078a95e72b7a
I20260806 04:39:27.486915 1585 tablet_service.cc:1514] Processing DeleteTablet for tablet cef7ceac37e440fdb7d2078a95e72b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:27 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:60200
I20260806 04:39:27.486969 1422 tablet_service.cc:1514] Processing DeleteTablet for tablet cef7ceac37e440fdb7d2078a95e72b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:27 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:46974
I20260806 04:39:27.487183 1287 tablet_service.cc:1514] Processing DeleteTablet for tablet cef7ceac37e440fdb7d2078a95e72b7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:27 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:35422
I20260806 04:39:27.487659 1705 tablet_replica.cc:330] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107: stopping tablet replica
I20260806 04:39:27.487699 1703 tablet_replica.cc:330] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21: stopping tablet replica
I20260806 04:39:27.487687 1136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:39:27.488143 1136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:39:27.488271 1703 raft_consensus.cc:2229] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:27.488328 1705 raft_consensus.cc:2229] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:39:27.488634 1703 raft_consensus.cc:2258] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:27.488890 1705 raft_consensus.cc:2258] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:27.489008 1704 tablet_replica.cc:330] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16: stopping tablet replica
I20260806 04:39:27.489279 1704 raft_consensus.cc:2229] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:27.490499 1705 ts_tablet_manager.cc:1905] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:27.490563 1703 ts_tablet_manager.cc:1905] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:27.493482 1703 ts_tablet_manager.cc:1918] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:39:27.493587 1703 log.cc:1196] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/cef7ceac37e440fdb7d2078a95e72b7a
I20260806 04:39:27.493508 1705 ts_tablet_manager.cc:1918] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:39:27.493794 1705 log.cc:1196] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/wal/wals/cef7ceac37e440fdb7d2078a95e72b7a
I20260806 04:39:27.493983 1703 ts_tablet_manager.cc:1939] T cef7ceac37e440fdb7d2078a95e72b7a P 2aa3f2eb048f4fe2b5344cc39ec27a21: Deleting consensus metadata
I20260806 04:39:27.494048 1287 tablet_service.cc:1467] Processing CreateTablet for tablet 33fd7f9e12d4444c8daa5391f475b27d (DEFAULT_TABLE table=test-table [id=da8ae8a35592491aace2201dc1bb18d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:27.494349 1287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33fd7f9e12d4444c8daa5391f475b27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:27.494360 1705 ts_tablet_manager.cc:1939] T cef7ceac37e440fdb7d2078a95e72b7a P a63bc75001934e98a9c827ee83d10107: Deleting consensus metadata
I20260806 04:39:27.495293 1123 catalog_manager.cc:4841] TS 2aa3f2eb048f4fe2b5344cc39ec27a21 (127.31.166.131:44573): tablet cef7ceac37e440fdb7d2078a95e72b7a (table test-table [id=a94b32e495bd42b788a99363b25d1f42]) successfully deleted
I20260806 04:39:27.494086 1422 tablet_service.cc:1467] Processing CreateTablet for tablet 33fd7f9e12d4444c8daa5391f475b27d (DEFAULT_TABLE table=test-table [id=da8ae8a35592491aace2201dc1bb18d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:27.495687 1422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33fd7f9e12d4444c8daa5391f475b27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:27.495693 1123 catalog_manager.cc:4841] TS a63bc75001934e98a9c827ee83d10107 (127.31.166.129:39125): tablet cef7ceac37e440fdb7d2078a95e72b7a (table test-table [id=a94b32e495bd42b788a99363b25d1f42]) successfully deleted
I20260806 04:39:27.496506 1679 tablet_bootstrap.cc:492] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107: Bootstrap starting.
I20260806 04:39:27.496738 1585 tablet_service.cc:1467] Processing CreateTablet for tablet 33fd7f9e12d4444c8daa5391f475b27d (DEFAULT_TABLE table=test-table [id=da8ae8a35592491aace2201dc1bb18d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:27.496991 1585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33fd7f9e12d4444c8daa5391f475b27d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:27.497488 1679 tablet_bootstrap.cc:654] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:27.498023 1680 tablet_bootstrap.cc:492] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16: Bootstrap starting.
I20260806 04:39:27.498936 1680 tablet_bootstrap.cc:654] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:27.498988 1679 tablet_bootstrap.cc:492] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107: No bootstrap required, opened a new log
I20260806 04:39:27.499071 1679 ts_tablet_manager.cc:1397] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:39:27.499442 1682 tablet_bootstrap.cc:492] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21: Bootstrap starting.
I20260806 04:39:27.499609 1679 raft_consensus.cc:348] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.499724 1679 raft_consensus.cc:374] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:27.499758 1679 raft_consensus.cc:729] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a63bc75001934e98a9c827ee83d10107, State: Initialized, Role: FOLLOWER
I20260806 04:39:27.499884 1679 consensus_queue.cc:260] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.499987 1704 raft_consensus.cc:2258] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:27.500159 1679 ts_tablet_manager.cc:1428] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 04:39:27.500406 1682 tablet_bootstrap.cc:654] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:27.501526 1704 ts_tablet_manager.cc:1905] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:27.502257 1682 tablet_bootstrap.cc:492] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21: No bootstrap required, opened a new log
I20260806 04:39:27.502341 1682 ts_tablet_manager.cc:1397] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:39:27.502588 1680 tablet_bootstrap.cc:492] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16: No bootstrap required, opened a new log
I20260806 04:39:27.502669 1680 ts_tablet_manager.cc:1397] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260806 04:39:27.502832 1682 raft_consensus.cc:348] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.502957 1682 raft_consensus.cc:374] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:27.503001 1682 raft_consensus.cc:729] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa3f2eb048f4fe2b5344cc39ec27a21, State: Initialized, Role: FOLLOWER
I20260806 04:39:27.503170 1682 consensus_queue.cc:260] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.503163 1680 raft_consensus.cc:348] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.503273 1680 raft_consensus.cc:374] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:27.503324 1680 raft_consensus.cc:729] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbd396205d3147fca269a2c4f8b0ed16, State: Initialized, Role: FOLLOWER
I20260806 04:39:27.503438 1682 ts_tablet_manager.cc:1428] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 04:39:27.503446 1680 consensus_queue.cc:260] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.503695 1680 ts_tablet_manager.cc:1428] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:39:27.504308 1704 ts_tablet_manager.cc:1918] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:39:27.504410 1704 log.cc:1196] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/cef7ceac37e440fdb7d2078a95e72b7a
I20260806 04:39:27.504701 1704 ts_tablet_manager.cc:1939] T cef7ceac37e440fdb7d2078a95e72b7a P cbd396205d3147fca269a2c4f8b0ed16: Deleting consensus metadata
I20260806 04:39:27.505836 1125 catalog_manager.cc:4841] TS cbd396205d3147fca269a2c4f8b0ed16 (127.31.166.130:41115): tablet cef7ceac37e440fdb7d2078a95e72b7a (table test-table [id=a94b32e495bd42b788a99363b25d1f42]) successfully deleted
I20260806 04:39:27.649590 1691 raft_consensus.cc:482] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:39:27.649745 1691 raft_consensus.cc:504] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.650192 1691 leader_election.cc:288] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbd396205d3147fca269a2c4f8b0ed16 (127.31.166.130:41115), 2aa3f2eb048f4fe2b5344cc39ec27a21 (127.31.166.131:44573)
I20260806 04:39:27.650727 1457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33fd7f9e12d4444c8daa5391f475b27d" candidate_uuid: "a63bc75001934e98a9c827ee83d10107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbd396205d3147fca269a2c4f8b0ed16" is_pre_election: true
I20260806 04:39:27.650730 1608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33fd7f9e12d4444c8daa5391f475b27d" candidate_uuid: "a63bc75001934e98a9c827ee83d10107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" is_pre_election: true
I20260806 04:39:27.650913 1608 raft_consensus.cc:2454] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a63bc75001934e98a9c827ee83d10107 in term 0.
I20260806 04:39:27.650913 1457 raft_consensus.cc:2454] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a63bc75001934e98a9c827ee83d10107 in term 0.
I20260806 04:39:27.651335 1235 leader_election.cc:302] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa3f2eb048f4fe2b5344cc39ec27a21, a63bc75001934e98a9c827ee83d10107; no voters:
I20260806 04:39:27.651525 1691 raft_consensus.cc:2781] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:39:27.651587 1691 raft_consensus.cc:482] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:39:27.651631 1691 raft_consensus.cc:3037] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:27.652618 1691 raft_consensus.cc:504] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.652949 1691 leader_election.cc:288] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [CANDIDATE]: Term 1 election: Requested vote from peers cbd396205d3147fca269a2c4f8b0ed16 (127.31.166.130:41115), 2aa3f2eb048f4fe2b5344cc39ec27a21 (127.31.166.131:44573)
I20260806 04:39:27.653368 1608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33fd7f9e12d4444c8daa5391f475b27d" candidate_uuid: "a63bc75001934e98a9c827ee83d10107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21"
I20260806 04:39:27.653513 1457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33fd7f9e12d4444c8daa5391f475b27d" candidate_uuid: "a63bc75001934e98a9c827ee83d10107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbd396205d3147fca269a2c4f8b0ed16"
I20260806 04:39:27.653524 1608 raft_consensus.cc:3037] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:27.653656 1457 raft_consensus.cc:3037] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:27.654572 1457 raft_consensus.cc:2454] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a63bc75001934e98a9c827ee83d10107 in term 1.
I20260806 04:39:27.654704 1608 raft_consensus.cc:2454] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a63bc75001934e98a9c827ee83d10107 in term 1.
I20260806 04:39:27.654901 1237 leader_election.cc:302] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a63bc75001934e98a9c827ee83d10107, cbd396205d3147fca269a2c4f8b0ed16; no voters:
I20260806 04:39:27.655144 1691 raft_consensus.cc:2781] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:27.655261 1691 raft_consensus.cc:686] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [term 1 LEADER]: Becoming Leader. State: Replica: a63bc75001934e98a9c827ee83d10107, State: Running, Role: LEADER
I20260806 04:39:27.655431 1691 consensus_queue.cc:237] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.657361 1136 catalog_manager.cc:5495] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 reported cstate change: term changed from 0 to 1, leader changed from <none> to a63bc75001934e98a9c827ee83d10107 (127.31.166.129). New cstate: current_term: 1 leader_uuid: "a63bc75001934e98a9c827ee83d10107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } health_report { overall_health: UNKNOWN } } }
Aug 06 04:39:27 dist-test-slave-vkjp krb5kdc[1091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991166, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:27.730392 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.717417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.31.166.190:33615)
0806 04:39:27.717601 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:27.717608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:27.717723 (+ 115us) server_negotiation.cc:407] Connection header received
0806 04:39:27.717835 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:27.717838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:27.717896 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:27.717984 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:27.718719 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.719387 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.720255 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.720496 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.720576 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.727130 (+ 6554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:27.727135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:27.727137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:27.727164 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:27.728659 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:27.729157 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:27.729160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:27.729162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:27.729239 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:27.729518 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:27.729521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:27.729522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:27.729732 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:27.729811 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:27.730133 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 04:39:27.730253 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":86}
I20260806 04:39:27.730432 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.717301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33615 (local address 127.0.0.1:48318)
0806 04:39:27.717621 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:27.717634 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:27.717743 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:27.718028 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:27.718030 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:27.718484 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:27.718586 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.718592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.719520 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.719523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.720142 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.720151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.720627 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.720630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.720699 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.725020 (+ 4321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:27.725037 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:27.726878 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:27.728782 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:27.728791 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:27.728797 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:27.729065 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:27.729322 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:27.729325 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:27.729326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:27.729420 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:27.729833 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:27.729841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:27.729961 (+ 120us) client_negotiation.cc:769] Sending connection context
0806 04:39:27.730062 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 04:39:27.730193 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":105,"threads_started":1}
I20260806 04:39:27.734928 1136 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:48318:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 04:39:27.735723 1136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260806 04:39:27.740768 1360 tablet.cc:2369] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:39:27.741524 1422 tablet_service.cc:1467] Processing CreateTablet for tablet 90d1ea45aa294bdc8144f42b9eab8255 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2e324c362f9040fa984258a676d7b5fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:39:27.741597 1585 tablet_service.cc:1467] Processing CreateTablet for tablet 90d1ea45aa294bdc8144f42b9eab8255 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2e324c362f9040fa984258a676d7b5fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:39:27.741958 1585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d1ea45aa294bdc8144f42b9eab8255. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:27.741989 1422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d1ea45aa294bdc8144f42b9eab8255. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:27.743664 1287 tablet_service.cc:1467] Processing CreateTablet for tablet 90d1ea45aa294bdc8144f42b9eab8255 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2e324c362f9040fa984258a676d7b5fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:39:27.744143 1287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d1ea45aa294bdc8144f42b9eab8255. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:27.745014 1680 tablet_bootstrap.cc:492] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16: Bootstrap starting.
I20260806 04:39:27.746160 1680 tablet_bootstrap.cc:654] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:27.746973 1682 tablet_bootstrap.cc:492] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21: Bootstrap starting.
I20260806 04:39:27.747346 1679 tablet_bootstrap.cc:492] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107: Bootstrap starting.
I20260806 04:39:27.747574 1680 tablet_bootstrap.cc:492] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16: No bootstrap required, opened a new log
I20260806 04:39:27.747681 1680 ts_tablet_manager.cc:1397] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260806 04:39:27.747928 1682 tablet_bootstrap.cc:654] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:27.748163 1680 raft_consensus.cc:348] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.748265 1679 tablet_bootstrap.cc:654] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:27.748319 1680 raft_consensus.cc:374] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:27.748368 1680 raft_consensus.cc:729] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbd396205d3147fca269a2c4f8b0ed16, State: Initialized, Role: FOLLOWER
I20260806 04:39:27.748515 1680 consensus_queue.cc:260] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.748842 1680 ts_tablet_manager.cc:1428] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260806 04:39:27.749143 1687 tablet_replica.cc:439] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } }
I20260806 04:39:27.749320 1687 tablet_replica.cc:442] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:39:27.749895 1679 tablet_bootstrap.cc:492] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107: No bootstrap required, opened a new log
I20260806 04:39:27.750016 1679 ts_tablet_manager.cc:1397] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260806 04:39:27.750412 1679 raft_consensus.cc:348] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.750540 1679 raft_consensus.cc:374] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:27.750586 1679 raft_consensus.cc:729] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a63bc75001934e98a9c827ee83d10107, State: Initialized, Role: FOLLOWER
I20260806 04:39:27.750771 1679 consensus_queue.cc:260] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.751060 1691 tablet_replica.cc:439] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } }
I20260806 04:39:27.751212 1691 tablet_replica.cc:442] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:39:27.751734 1679 ts_tablet_manager.cc:1428] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 04:39:27.752691 1682 tablet_bootstrap.cc:492] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21: No bootstrap required, opened a new log
I20260806 04:39:27.752769 1682 ts_tablet_manager.cc:1397] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260806 04:39:27.753186 1682 raft_consensus.cc:348] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.753281 1682 raft_consensus.cc:374] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:27.753309 1682 raft_consensus.cc:729] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa3f2eb048f4fe2b5344cc39ec27a21, State: Initialized, Role: FOLLOWER
I20260806 04:39:27.753429 1682 consensus_queue.cc:260] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.753660 1688 tablet_replica.cc:439] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } }
I20260806 04:39:27.753878 1688 tablet_replica.cc:442] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:39:27.753675 1682 ts_tablet_manager.cc:1428] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 04:39:27.901522 1687 raft_consensus.cc:482] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:39:27.901681 1687 raft_consensus.cc:504] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.902964 1687 leader_election.cc:288] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a63bc75001934e98a9c827ee83d10107 (127.31.166.129:39125), 2aa3f2eb048f4fe2b5344cc39ec27a21 (127.31.166.131:44573)
I20260806 04:39:27.907196 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.903390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:39125 (local address 127.31.166.130:47507)
0806 04:39:27.903785 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:27.903801 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:27.903976 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:27.904252 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:27.904256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:27.904276 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:27.904393 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.904399 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.905147 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.905150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.905994 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.906003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.906688 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.906692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.906789 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.906817 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 04:39:27.906946 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 04:39:27.907023 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":105,"threads_started":1}
I20260806 04:39:27.907184 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.903466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:47507 (local address 127.31.166.129:39125)
0806 04:39:27.903632 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:27.903639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:27.903890 (+ 251us) server_negotiation.cc:407] Connection header received
0806 04:39:27.904095 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:27.904098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:27.904154 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:27.904239 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:27.904503 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.905019 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.906123 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.906532 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.906699 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.906764 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:27.906969 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 04:39:27.907046 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:39:27.907905 1308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d1ea45aa294bdc8144f42b9eab8255" candidate_uuid: "cbd396205d3147fca269a2c4f8b0ed16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a63bc75001934e98a9c827ee83d10107" is_pre_election: true
I20260806 04:39:27.908166 1308 raft_consensus.cc:2454] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbd396205d3147fca269a2c4f8b0ed16 in term 0.
I20260806 04:39:27.908222 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.904013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:59515 (local address 127.31.166.131:44573)
0806 04:39:27.904160 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:27.904166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:27.904396 (+ 230us) server_negotiation.cc:407] Connection header received
0806 04:39:27.904571 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:27.904575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:27.904630 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:27.904725 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:27.905066 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.905630 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.906905 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.907590 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.907756 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.907821 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:27.908013 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 04:39:27.908090 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:39:27.908254 1721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.903871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:44573 (local address 127.31.166.130:59515)
0806 04:39:27.904290 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:27.904305 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:27.904447 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:27.904757 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:27.904760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:27.904779 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:27.904931 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.904938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.905759 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.905764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.906744 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.906759 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.907732 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.907737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.907867 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.907904 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 04:39:27.907996 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 04:39:27.908072 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":69,"threads_started":1}
I20260806 04:39:27.908653 1384 leader_election.cc:302] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a63bc75001934e98a9c827ee83d10107, cbd396205d3147fca269a2c4f8b0ed16; no voters:
I20260806 04:39:27.908715 1608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d1ea45aa294bdc8144f42b9eab8255" candidate_uuid: "cbd396205d3147fca269a2c4f8b0ed16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" is_pre_election: true
I20260806 04:39:27.908906 1608 raft_consensus.cc:2454] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbd396205d3147fca269a2c4f8b0ed16 in term 0.
I20260806 04:39:27.908933 1687 raft_consensus.cc:2781] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:39:27.909000 1687 raft_consensus.cc:482] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:39:27.909062 1687 raft_consensus.cc:3037] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:27.910295 1687 raft_consensus.cc:504] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.910686 1687 leader_election.cc:288] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [CANDIDATE]: Term 1 election: Requested vote from peers a63bc75001934e98a9c827ee83d10107 (127.31.166.129:39125), 2aa3f2eb048f4fe2b5344cc39ec27a21 (127.31.166.131:44573)
I20260806 04:39:27.911150 1308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d1ea45aa294bdc8144f42b9eab8255" candidate_uuid: "cbd396205d3147fca269a2c4f8b0ed16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a63bc75001934e98a9c827ee83d10107"
I20260806 04:39:27.911196 1608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d1ea45aa294bdc8144f42b9eab8255" candidate_uuid: "cbd396205d3147fca269a2c4f8b0ed16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21"
I20260806 04:39:27.911316 1608 raft_consensus.cc:3037] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:27.911316 1308 raft_consensus.cc:3037] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:27.912654 1608 raft_consensus.cc:2454] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbd396205d3147fca269a2c4f8b0ed16 in term 1.
I20260806 04:39:27.912663 1308 raft_consensus.cc:2454] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbd396205d3147fca269a2c4f8b0ed16 in term 1.
I20260806 04:39:27.913134 1384 leader_election.cc:302] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2aa3f2eb048f4fe2b5344cc39ec27a21, cbd396205d3147fca269a2c4f8b0ed16; no voters:
I20260806 04:39:27.913389 1687 raft_consensus.cc:2781] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:27.913594 1687 raft_consensus.cc:686] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [term 1 LEADER]: Becoming Leader. State: Replica: cbd396205d3147fca269a2c4f8b0ed16, State: Running, Role: LEADER
I20260806 04:39:27.913849 1687 consensus_queue.cc:237] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } }
I20260806 04:39:27.915271 1722 tablet_replica.cc:439] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16: TxnStatusTablet state changed. Reason: New leader cbd396205d3147fca269a2c4f8b0ed16. Latest consensus state: current_term: 1 leader_uuid: "cbd396205d3147fca269a2c4f8b0ed16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } }
I20260806 04:39:27.915390 1722 tablet_replica.cc:442] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:39:27.915709 1724 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 04:39:27.915903 1724 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 04:39:27.916558 1136 catalog_manager.cc:5495] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbd396205d3147fca269a2c4f8b0ed16 (127.31.166.130). New cstate: current_term: 1 leader_uuid: "cbd396205d3147fca269a2c4f8b0ed16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } health_report { overall_health: UNKNOWN } } }
I20260806 04:39:27.951411 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.946730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:41115 (local address 127.0.0.1:46988)
0806 04:39:27.946915 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:27.946934 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:27.947060 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:27.947549 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:27.947551 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:27.947558 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:39:27.947655 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.947659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.948834 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.948836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.949447 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.949457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.950052 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.950055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.950129 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.950175 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:39:27.951055 (+ 880us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:39:27.951072 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 04:39:27.951197 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 04:39:27.951275 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 04:39:27.951473 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.946883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.31.166.130:41115)
0806 04:39:27.947225 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:27.947238 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:27.947254 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:39:27.947332 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:27.947335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:27.947410 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:27.947525 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:27.947776 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.948705 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.949562 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.949929 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.950035 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.950336 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:27.950928 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:27.951047 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:27.951216 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 04:39:27.951276 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":135,"threads_started":1}
I20260806 04:39:27.954644 1608 raft_consensus.cc:1261] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 1 FOLLOWER]: Refusing update from remote peer cbd396205d3147fca269a2c4f8b0ed16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:27.954824 1308 raft_consensus.cc:1261] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107 [term 1 FOLLOWER]: Refusing update from remote peer cbd396205d3147fca269a2c4f8b0ed16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:27.955221 1687 consensus_queue.cc:1035] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:27.955502 1722 consensus_queue.cc:1035] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:27.956952 1688 tablet_replica.cc:439] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21: TxnStatusTablet state changed. Reason: New leader cbd396205d3147fca269a2c4f8b0ed16. Latest consensus state: current_term: 1 leader_uuid: "cbd396205d3147fca269a2c4f8b0ed16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } }
I20260806 04:39:27.957075 1688 tablet_replica.cc:442] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:39:27.957333 1691 tablet_replica.cc:439] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107: TxnStatusTablet state changed. Reason: New leader cbd396205d3147fca269a2c4f8b0ed16. Latest consensus state: current_term: 1 leader_uuid: "cbd396205d3147fca269a2c4f8b0ed16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } }
I20260806 04:39:27.957499 1691 tablet_replica.cc:442] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:39:27.959587 1722 tablet_replica.cc:439] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cbd396205d3147fca269a2c4f8b0ed16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } }
I20260806 04:39:27.959738 1722 tablet_replica.cc:442] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:39:27.961086 1688 tablet_replica.cc:439] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cbd396205d3147fca269a2c4f8b0ed16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } }
I20260806 04:39:27.961202 1688 tablet_replica.cc:442] T 90d1ea45aa294bdc8144f42b9eab8255 P 2aa3f2eb048f4fe2b5344cc39ec27a21: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:39:27.964392 1687 tablet_replica.cc:439] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cbd396205d3147fca269a2c4f8b0ed16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } }
I20260806 04:39:27.964519 1687 tablet_replica.cc:442] T 90d1ea45aa294bdc8144f42b9eab8255 P cbd396205d3147fca269a2c4f8b0ed16: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:39:27.965356 1691 tablet_replica.cc:439] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cbd396205d3147fca269a2c4f8b0ed16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a63bc75001934e98a9c827ee83d10107" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39125 } } peers { permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 } } peers { permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 } } }
I20260806 04:39:27.965471 1691 tablet_replica.cc:442] T 90d1ea45aa294bdc8144f42b9eab8255 P a63bc75001934e98a9c827ee83d10107: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:39:27.975760 1727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.971390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:54207 (local address 127.31.166.130:41115)
0806 04:39:27.971517 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:27.971521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:27.971770 (+ 249us) server_negotiation.cc:407] Connection header received
0806 04:39:27.971971 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:27.971974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:27.972023 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:27.972065 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:27.972414 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.973221 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.974531 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.974896 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.975015 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.975320 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:27.975401 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:27.975468 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:27.975584 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 04:39:27.975638 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 04:39:27.975800 1734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:27.971316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:41115 (local address 127.31.166.129:54207)
0806 04:39:27.971691 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:27.971706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:27.971859 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:27.972186 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:27.972189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:27.972196 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:39:27.972299 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.972305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.973374 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.973378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.974423 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:27.974434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:27.975016 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:27.975020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:27.975134 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:27.975185 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:39:27.975492 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:39:27.975499 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:39:27.975566 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 04:39:27.975647 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
I20260806 04:39:27.983137 1608 raft_consensus.cc:1261] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 1 FOLLOWER]: Refusing update from remote peer a63bc75001934e98a9c827ee83d10107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:27.983273 1457 raft_consensus.cc:1261] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16 [term 1 FOLLOWER]: Refusing update from remote peer a63bc75001934e98a9c827ee83d10107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:27.983579 1691 consensus_queue.cc:1035] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aa3f2eb048f4fe2b5344cc39ec27a21" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:27.983722 1686 consensus_queue.cc:1035] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbd396205d3147fca269a2c4f8b0ed16" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 41115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:28.016420 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:28.012276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:39125 (local address 127.31.166.130:53983)
0806 04:39:28.012633 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:28.012646 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:28.012734 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:28.012963 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:28.012965 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:28.012971 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:39:28.013081 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:28.013086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:28.014135 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:28.014139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:28.015071 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:28.015102 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:28.015721 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:28.015724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:28.015807 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:28.015857 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:39:28.016163 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:39:28.016168 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:39:28.016220 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 04:39:28.016298 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":136,"threads_started":1}
I20260806 04:39:28.016429 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:28.012363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:53983 (local address 127.31.166.129:39125)
0806 04:39:28.012496 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:28.012500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:28.012695 (+ 195us) server_negotiation.cc:407] Connection header received
0806 04:39:28.012816 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:28.012819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:28.012874 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:28.012954 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:28.013185 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:28.014006 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:28.015212 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:28.015560 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:28.015665 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:28.015993 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:28.016076 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:28.016151 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:28.016231 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 04:39:28.016291 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:39:28.042546 1137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:28.042781 1137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:28.045255 1137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb0fb22b06424059be33dfa84ba539a5: Sending DeleteTablet for 3 replicas of tablet 33fd7f9e12d4444c8daa5391f475b27d
I20260806 04:39:28.045920 1287 tablet_service.cc:1514] Processing DeleteTablet for tablet 33fd7f9e12d4444c8daa5391f475b27d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:28 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:35422
I20260806 04:39:28.045929 1422 tablet_service.cc:1514] Processing DeleteTablet for tablet 33fd7f9e12d4444c8daa5391f475b27d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:28 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:46974
I20260806 04:39:28.046274 1742 tablet_replica.cc:330] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107: stopping tablet replica
I20260806 04:39:28.046411 1742 raft_consensus.cc:2229] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:39:28.046561 1585 tablet_service.cc:1514] Processing DeleteTablet for tablet 33fd7f9e12d4444c8daa5391f475b27d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:28 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:60200
I20260806 04:39:28.046622 1742 raft_consensus.cc:2258] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:28.046895 1744 tablet_replica.cc:330] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16: stopping tablet replica
I20260806 04:39:28.047031 1744 raft_consensus.cc:2229] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:28.047049 1746 tablet_replica.cc:330] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21: stopping tablet replica
I20260806 04:39:28.047188 1744 raft_consensus.cc:2258] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:28.047601 1742 ts_tablet_manager.cc:1905] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:28.047608 1746 raft_consensus.cc:2229] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:28.047829 1746 raft_consensus.cc:2258] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:28.047900 1744 ts_tablet_manager.cc:1905] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:28.050278 1744 ts_tablet_manager.cc:1918] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:39:28.050377 1744 log.cc:1196] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/33fd7f9e12d4444c8daa5391f475b27d
I20260806 04:39:28.050637 1744 ts_tablet_manager.cc:1939] T 33fd7f9e12d4444c8daa5391f475b27d P cbd396205d3147fca269a2c4f8b0ed16: Deleting consensus metadata
I20260806 04:39:28.050859 1742 ts_tablet_manager.cc:1918] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:39:28.050954 1742 log.cc:1196] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-0/wal/wals/33fd7f9e12d4444c8daa5391f475b27d
I20260806 04:39:28.051285 1742 ts_tablet_manager.cc:1939] T 33fd7f9e12d4444c8daa5391f475b27d P a63bc75001934e98a9c827ee83d10107: Deleting consensus metadata
I20260806 04:39:28.051616 1125 catalog_manager.cc:4841] TS cbd396205d3147fca269a2c4f8b0ed16 (127.31.166.130:41115): tablet 33fd7f9e12d4444c8daa5391f475b27d (table test-table [id=da8ae8a35592491aace2201dc1bb18d6]) successfully deleted
I20260806 04:39:28.051918 1746 ts_tablet_manager.cc:1905] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:28.051925 1123 catalog_manager.cc:4841] TS a63bc75001934e98a9c827ee83d10107 (127.31.166.129:39125): tablet 33fd7f9e12d4444c8daa5391f475b27d (table test-table [id=da8ae8a35592491aace2201dc1bb18d6]) successfully deleted
I20260806 04:39:28.054081 1746 ts_tablet_manager.cc:1918] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:39:28.054147 1746 log.cc:1196] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/33fd7f9e12d4444c8daa5391f475b27d
I20260806 04:39:28.054412 1746 ts_tablet_manager.cc:1939] T 33fd7f9e12d4444c8daa5391f475b27d P 2aa3f2eb048f4fe2b5344cc39ec27a21: Deleting consensus metadata
I20260806 04:39:28.055141 1123 catalog_manager.cc:4841] TS 2aa3f2eb048f4fe2b5344cc39ec27a21 (127.31.166.131:44573): tablet 33fd7f9e12d4444c8daa5391f475b27d (table test-table [id=da8ae8a35592491aace2201dc1bb18d6]) successfully deleted
Aug 06 04:39:28 dist-test-slave-vkjp krb5kdc[1091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991167, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.166.129@KRBTEST.COM
I20260806 04:39:28.065203 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:28.054611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.31.166.129:39125)
0806 04:39:28.054737 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:28.054742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:28.055129 (+ 387us) server_negotiation.cc:407] Connection header received
0806 04:39:28.055360 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:28.055363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:28.055414 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:28.055484 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:28.056362 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:28.056905 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:28.058108 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:28.058316 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:28.058401 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:28.061492 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:28.061505 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:28.061522 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:28.061592 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:28.063407 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:28.063851 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:28.063857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:28.063858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:28.063943 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:28.064242 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:28.064247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:28.064249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:28.064504 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:28.064715 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:28.064894 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 04:39:28.065018 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 04:39:28.065660 1351 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35450
I20260806 04:39:28.076419 1191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:28.067375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.31.166.190:33615)
0806 04:39:28.067512 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:28.067515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:28.067605 (+ 90us) server_negotiation.cc:407] Connection header received
0806 04:39:28.067743 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:28.067746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:28.067785 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:28.067849 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:28.068631 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:28.069147 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:28.070260 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:28.070468 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:28.070543 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:28.072360 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:28.072364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:28.072366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:28.072406 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:28.074393 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:28.074931 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:28.074934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:28.074936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:28.075031 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:28.075454 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:28.075457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:28.075458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:28.075697 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:28.075819 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:28.076112 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 04:39:28.076229 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 04:39:28.076876 1137 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48334
I20260806 04:39:28.078071 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 1217
I20260806 04:39:28.086529 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 1368
I20260806 04:39:28.094345 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 1517
I20260806 04:39:28.101934 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 1105
2026-08-06T04:39:28Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3056 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:39:28 dist-test-slave-vkjp krb5kdc[1753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:39:28 dist-test-slave-vkjp krb5kdc[1753](info): set up 2 sockets
Aug 06 04:39:28 dist-test-slave-vkjp krb5kdc[1753](info): commencing operation
krb5kdc: starting...
W20260806 04:39:29.174665 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:39:29 dist-test-slave-vkjp krb5kdc[1753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:39:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:39:29.298846 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:45297
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:45297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:29.412967 1768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:29.413283 1768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:29.413388 1768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:29.417465 1768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:29.417557 1768 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 04:39:29.417588 1768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:29.417632 1768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:29.417670 1768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:29.417699 1768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:39:29.423161 1768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:45297
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:29.424299 1768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:29.425340 1768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:29.434165 1777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:29.434305 1768 server_base.cc:1034] running on GCE node
W20260806 04:39:29.434197 1774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:29.434561 1775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:29.435245 1768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:29.436354 1768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:29.437705 1768 hybrid_clock.cc:648] HybridClock initialized: now 1785991169437688 us; error 32 us; skew 500 ppm
Aug 06 04:39:29 dist-test-slave-vkjp krb5kdc[1753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991169, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:29.440652 1768 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:39:29.441550 1768 webserver.cc:460] Webserver started at http://127.31.166.190:43219/ using document root <none> and password file <none>
I20260806 04:39:29.442112 1768 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:29.442231 1768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:29.442487 1768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:29.444670 1768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "f20edfa9e07c40579859dd5d8cebea06"
format_stamp: "Formatted at 2026-08-06 04:39:29 on dist-test-slave-vkjp"
server_key: "957761f4f18fa9acd4ea2f313caf042b"
server_key_iv: "5170132b9cc09629e77dce3dfeaba51c"
server_key_version: "encryptionkey@0"
I20260806 04:39:29.445251 1768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "f20edfa9e07c40579859dd5d8cebea06"
format_stamp: "Formatted at 2026-08-06 04:39:29 on dist-test-slave-vkjp"
server_key: "957761f4f18fa9acd4ea2f313caf042b"
server_key_iv: "5170132b9cc09629e77dce3dfeaba51c"
server_key_version: "encryptionkey@0"
I20260806 04:39:29.448913 1768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:39:29.451210 1784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:29.452389 1768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:39:29.452524 1768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "f20edfa9e07c40579859dd5d8cebea06"
format_stamp: "Formatted at 2026-08-06 04:39:29 on dist-test-slave-vkjp"
server_key: "957761f4f18fa9acd4ea2f313caf042b"
server_key_iv: "5170132b9cc09629e77dce3dfeaba51c"
server_key_version: "encryptionkey@0"
I20260806 04:39:29.452633 1768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:29.468628 1768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:29.471526 1768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:29.471748 1768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:29.479794 1768 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:45297
I20260806 04:39:29.479863 1855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:45297 every 8 connection(s)
I20260806 04:39:29.480753 1768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:29.483497 1856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:29.485600 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 1768
I20260806 04:39:29.485726 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:29.486006 32410 external_mini_cluster.cc:1428] Setting key bf5d4bdedba58386fec0051b16852e01
I20260806 04:39:29.489241 1856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06: Bootstrap starting.
I20260806 04:39:29.491773 1856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:29.492587 1856 log.cc:824] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:29.494879 1856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06: No bootstrap required, opened a new log
Aug 06 04:39:29 dist-test-slave-vkjp krb5kdc[1753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991169, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:29.499315 1856 raft_consensus.cc:348] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f20edfa9e07c40579859dd5d8cebea06" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45297 } }
I20260806 04:39:29.499576 1856 raft_consensus.cc:374] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:29.499646 1856 raft_consensus.cc:729] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f20edfa9e07c40579859dd5d8cebea06, State: Initialized, Role: FOLLOWER
I20260806 04:39:29.500172 1856 consensus_queue.cc:260] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f20edfa9e07c40579859dd5d8cebea06" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45297 } }
I20260806 04:39:29.500342 1856 raft_consensus.cc:388] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:29.500470 1856 raft_consensus.cc:482] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:29.500597 1856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:29.501657 1856 raft_consensus.cc:504] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f20edfa9e07c40579859dd5d8cebea06" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45297 } }
I20260806 04:39:29.501951 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:29.487430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53180 (local address 127.31.166.190:45297)
0806 04:39:29.487827 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:29.487841 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:29.487875 (+ 34us) server_negotiation.cc:407] Connection header received
0806 04:39:29.488475 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:29.488490 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:29.488793 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:29.489062 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:29.489977 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.490979 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.492145 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.492690 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.492947 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:29.495739 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:29.495745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:29.495761 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:29.495793 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:29.498600 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:29.499064 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:29.499070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:29.499075 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:29.499235 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:29.499729 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:29.499733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:29.499735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:29.500218 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:29.500436 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:29.500818 (+ 382us) server_negotiation.cc:299] Negotiation successful
0806 04:39:29.501106 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":136,"threads_started":1}
I20260806 04:39:29.502046 1856 leader_election.cc:302] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f20edfa9e07c40579859dd5d8cebea06; no voters:
I20260806 04:39:29.502437 1856 leader_election.cc:288] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:29.502589 1863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:29.502969 1863 raft_consensus.cc:686] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [term 1 LEADER]: Becoming Leader. State: Replica: f20edfa9e07c40579859dd5d8cebea06, State: Running, Role: LEADER
I20260806 04:39:29.503355 1863 consensus_queue.cc:237] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f20edfa9e07c40579859dd5d8cebea06" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45297 } }
I20260806 04:39:29.503697 1856 sys_catalog.cc:564] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:29.505295 1865 sys_catalog.cc:455] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f20edfa9e07c40579859dd5d8cebea06. Latest consensus state: current_term: 1 leader_uuid: "f20edfa9e07c40579859dd5d8cebea06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f20edfa9e07c40579859dd5d8cebea06" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45297 } } }
I20260806 04:39:29.505267 1864 sys_catalog.cc:455] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f20edfa9e07c40579859dd5d8cebea06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f20edfa9e07c40579859dd5d8cebea06" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45297 } } }
I20260806 04:39:29.505403 1865 sys_catalog.cc:458] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:29.505403 1864 sys_catalog.cc:458] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:29.505844 1875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:29.508597 1875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:29.513949 1875 catalog_manager.cc:1347] Generated new cluster ID: cfd1e127d608424185a7cf77b4fd7263
I20260806 04:39:29.514015 1875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:29.523367 1875 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:29.524366 1875 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:29.531585 1875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06: Generated new TSK 0
I20260806 04:39:29.532244 1875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:39:29.562124 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:45297
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:29.675694 1886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:29.675947 1886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:29.676023 1886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:29.680114 1886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:29.680167 1886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:29.680230 1886 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 04:39:29.680258 1886 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:39:29.680317 1886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:39:29.685429 1886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.166.190:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:29.686460 1886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:29.687494 1886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:29.697155 1893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:29.697220 1897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:29.697207 1894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:29.697456 1886 server_base.cc:1034] running on GCE node
I20260806 04:39:29.697842 1886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:29.698462 1886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:29.699623 1886 hybrid_clock.cc:648] HybridClock initialized: now 1785991169699607 us; error 35 us; skew 500 ppm
Aug 06 04:39:29 dist-test-slave-vkjp krb5kdc[1753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991169, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:29.702484 1886 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:39:29.703431 1886 webserver.cc:460] Webserver started at http://127.31.166.129:42785/ using document root <none> and password file <none>
I20260806 04:39:29.704020 1886 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:29.704128 1886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:29.704375 1886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:29.706573 1886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "964cb62944bc40daadba7dadcf2ad840"
format_stamp: "Formatted at 2026-08-06 04:39:29 on dist-test-slave-vkjp"
server_key: "314fd9b9933c8ee62a091b282cedff25"
server_key_iv: "21c05f194018f92b607b476d30e4ef80"
server_key_version: "encryptionkey@0"
I20260806 04:39:29.707223 1886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "964cb62944bc40daadba7dadcf2ad840"
format_stamp: "Formatted at 2026-08-06 04:39:29 on dist-test-slave-vkjp"
server_key: "314fd9b9933c8ee62a091b282cedff25"
server_key_iv: "21c05f194018f92b607b476d30e4ef80"
server_key_version: "encryptionkey@0"
I20260806 04:39:29.710770 1886 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:39:29.713168 1904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:29.714354 1886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:39:29.714483 1886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "964cb62944bc40daadba7dadcf2ad840"
format_stamp: "Formatted at 2026-08-06 04:39:29 on dist-test-slave-vkjp"
server_key: "314fd9b9933c8ee62a091b282cedff25"
server_key_iv: "21c05f194018f92b607b476d30e4ef80"
server_key_version: "encryptionkey@0"
I20260806 04:39:29.714589 1886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:29.726480 1886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:29.729480 1886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:29.729677 1886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:29.731154 1886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:29.731233 1886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:29.731303 1886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:29.731359 1886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:39:29 dist-test-slave-vkjp krb5kdc[1753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991169, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:29.742722 1886 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:45595
I20260806 04:39:29.742807 2025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:45595 every 8 connection(s)
I20260806 04:39:29.743891 1886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:39:29.745743 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:29.731882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:58627 (local address 127.31.166.190:45297)
0806 04:39:29.732013 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:29.732016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:29.733054 (+ 1038us) server_negotiation.cc:407] Connection header received
0806 04:39:29.733976 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:29.733980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:29.734042 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:29.734147 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:29.735610 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.736326 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.737475 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.737714 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.737837 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:29.741757 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:29.741761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:29.741763 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:29.741793 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:29.743628 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:29.744204 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:29.744210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:29.744215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:29.744380 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:29.744787 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:29.744790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:29.744791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:29.744979 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:29.745063 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:29.745512 (+ 449us) server_negotiation.cc:299] Negotiation successful
0806 04:39:29.745623 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:39:29.746491 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:29.732393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:45297 (local address 127.31.166.129:58627)
0806 04:39:29.732898 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:29.732951 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:29.733772 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:29.734377 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:29.734386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:29.734846 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:29.735461 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:29.735475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.736454 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.736457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:29.737334 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:29.737344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.737871 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.737875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:29.738062 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:29.738772 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:29.738795 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:29.741590 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:29.743744 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:29.743748 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:29.743759 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:29.744086 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:29.744507 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:29.744512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:29.744518 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:29.744695 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:29.745084 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:29.745089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:29.745302 (+ 213us) client_negotiation.cc:769] Sending connection context
0806 04:39:29.745461 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 04:39:29.745698 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":131,"threads_started":1}
I20260806 04:39:29.747807 2027 heartbeater.cc:347] Connected to a master server at 127.31.166.190:45297
I20260806 04:39:29.747949 2027 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:29.748512 2027 heartbeater.cc:511] Master 127.31.166.190:45297 requested a full tablet report, sending...
I20260806 04:39:29.748900 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 1886
I20260806 04:39:29.749037 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:39:29.749389 32410 external_mini_cluster.cc:1428] Setting key 1b65f393b916a4cc0023310206c7d50f
I20260806 04:39:29.750319 1802 ts_manager.cc:194] Registered new tserver with Master: 964cb62944bc40daadba7dadcf2ad840 (127.31.166.129:45595)
I20260806 04:39:29.751868 1802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:58627
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
I20260806 04:39:29.760836 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:29.750228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:45297 (local address 127.31.166.129:39741)
0806 04:39:29.750526 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:29.750542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:29.750632 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:29.752963 (+ 2331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:29.752968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:29.753223 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:29.753444 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:29.753455 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.754469 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.754472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:29.755158 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:29.755167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.755873 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.755880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:29.755962 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:29.756611 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:29.756632 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:29.757107 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:29.759248 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:29.759251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:29.759253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:29.759490 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:29.759817 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:29.759819 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:29.759821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:29.759906 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:29.760370 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:29.760378 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:29.760485 (+ 107us) client_negotiation.cc:769] Sending connection context
0806 04:39:29.760573 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 04:39:29.760685 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":1,"thread_start_us":105,"threads_started":1}
I20260806 04:39:29.760912 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:29.752492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:39741 (local address 127.31.166.190:45297)
0806 04:39:29.752617 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:29.752623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:29.752640 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:39:29.752684 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:29.752689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:29.752741 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:29.752837 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:29.753588 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.754320 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.755274 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.755681 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.755818 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:29.757255 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:29.757261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:29.757266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:29.757298 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:29.759138 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:29.759595 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:29.759599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:29.759603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:29.759706 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:29.760006 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:29.760012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:29.760015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:29.760227 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:29.760346 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:29.760673 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 04:39:29.760799 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:39:29.778891 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:45297
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:29.891021 2037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:29.891285 2037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:29.891331 2037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:29.895699 2037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:29.895753 2037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:29.895792 2037 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 04:39:29.895838 2037 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:39:29.895895 2037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:39:29.901041 2037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.166.190:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:29.902051 2037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:29.903028 2037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:29.912875 2043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:29.912900 2045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:29.913072 2037 server_base.cc:1034] running on GCE node
W20260806 04:39:29.912887 2042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:29.913539 2037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:29.914186 2037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:29.915360 2037 hybrid_clock.cc:648] HybridClock initialized: now 1785991169915336 us; error 35 us; skew 500 ppm
Aug 06 04:39:29 dist-test-slave-vkjp krb5kdc[1753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991169, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:29.917991 2037 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:39:29.918967 2037 webserver.cc:460] Webserver started at http://127.31.166.130:46589/ using document root <none> and password file <none>
I20260806 04:39:29.919621 2037 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:29.919726 2037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:29.919970 2037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:29.922003 2037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "6815f42fd2384e43aa7f714b43e3753a"
format_stamp: "Formatted at 2026-08-06 04:39:29 on dist-test-slave-vkjp"
server_key: "e2b8a187f60c9fda0a38bd92e3716636"
server_key_iv: "e93c55b40ed9770a3d2fa9cfe466104a"
server_key_version: "encryptionkey@0"
I20260806 04:39:29.922560 2037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "6815f42fd2384e43aa7f714b43e3753a"
format_stamp: "Formatted at 2026-08-06 04:39:29 on dist-test-slave-vkjp"
server_key: "e2b8a187f60c9fda0a38bd92e3716636"
server_key_iv: "e93c55b40ed9770a3d2fa9cfe466104a"
server_key_version: "encryptionkey@0"
I20260806 04:39:29.926280 2037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 04:39:29.928707 2055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:29.929893 2037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:39:29.930027 2037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "6815f42fd2384e43aa7f714b43e3753a"
format_stamp: "Formatted at 2026-08-06 04:39:29 on dist-test-slave-vkjp"
server_key: "e2b8a187f60c9fda0a38bd92e3716636"
server_key_iv: "e93c55b40ed9770a3d2fa9cfe466104a"
server_key_version: "encryptionkey@0"
I20260806 04:39:29.930137 2037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:29.939673 2037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:29.942553 2037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:29.942770 2037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:29.944437 2037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:29.944522 2037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:29.944595 2037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:29.944664 2037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:39:29 dist-test-slave-vkjp krb5kdc[1753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991169, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:29.956359 2037 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:44441
I20260806 04:39:29.956700 2176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:44441 every 8 connection(s)
I20260806 04:39:29.957381 2037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:39:29.959540 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:29.945104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:49085 (local address 127.31.166.190:45297)
0806 04:39:29.945241 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:29.945247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:29.946641 (+ 1394us) server_negotiation.cc:407] Connection header received
0806 04:39:29.947962 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:29.947967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:29.948039 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:29.948103 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:29.949798 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.950341 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.951810 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.952048 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.952164 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:29.955386 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:29.955389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:29.955392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:29.955416 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:29.957163 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:29.957772 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:29.957775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:29.957777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:29.957907 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:29.958365 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:29.958370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:29.958373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:29.958586 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:29.958685 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:29.959236 (+ 551us) server_negotiation.cc:299] Negotiation successful
0806 04:39:29.959414 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:39:29.960281 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:29.946041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:45297 (local address 127.31.166.130:49085)
0806 04:39:29.946487 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:29.946536 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:29.947679 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:29.948623 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:29.948632 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:29.949168 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:29.949647 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:29.949660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.950473 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.950479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:29.951672 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:29.951682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.952220 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.952224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:29.952408 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:29.953046 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:29.953063 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:29.955166 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:29.957299 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:29.957305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:29.957335 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:29.957664 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:29.958025 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:29.958029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:29.958031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:29.958208 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:29.958712 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:29.958716 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:29.958931 (+ 215us) client_negotiation.cc:769] Sending connection context
0806 04:39:29.959142 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 04:39:29.959464 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":138,"threads_started":1}
I20260806 04:39:29.961572 2177 heartbeater.cc:347] Connected to a master server at 127.31.166.190:45297
I20260806 04:39:29.961748 2177 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:29.962283 2177 heartbeater.cc:511] Master 127.31.166.190:45297 requested a full tablet report, sending...
I20260806 04:39:29.963368 1802 ts_manager.cc:194] Registered new tserver with Master: 6815f42fd2384e43aa7f714b43e3753a (127.31.166.130:44441)
I20260806 04:39:29.964182 1802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:49085
I20260806 04:39:29.965516 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 2037
I20260806 04:39:29.965626 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:39:29.965950 32410 external_mini_cluster.cc:1428] Setting key c8928baddc26b5f0201297b8c95b4c1c
I20260806 04:39:29.971938 2183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:29.963684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:45297 (local address 127.31.166.130:58431)
0806 04:39:29.963977 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:29.963989 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:29.964062 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:29.964771 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:29.964774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:29.964977 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:29.965080 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:29.965086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.965914 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.965917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:29.966748 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:29.966761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.967387 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.967391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:29.967487 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:29.968045 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:29.968059 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:29.968382 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:29.970211 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:29.970214 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:29.970216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:29.970504 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:29.970902 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:29.970905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:29.970906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:29.970984 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:29.971518 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:29.971520 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:29.971593 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 04:39:29.971678 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 04:39:29.971798 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":102,"threads_started":1}
I20260806 04:39:29.972123 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:29.964461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:58431 (local address 127.31.166.190:45297)
0806 04:39:29.964587 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:29.964590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:29.964601 (+ 11us) server_negotiation.cc:407] Connection header received
0806 04:39:29.964629 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:29.964631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:29.964664 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:29.964753 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:29.965195 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.965788 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.966893 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:29.967247 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:29.967363 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:29.968529 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:29.968535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:29.968541 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:29.968572 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:29.970085 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:29.970657 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:29.970660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:29.970682 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:29.970796 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:29.971078 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:29.971106 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:29.971109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:29.971417 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:29.971524 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:29.971763 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 04:39:29.971893 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:39:29.994375 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:45297
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:30.107488 2186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:30.107752 2186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:30.107854 2186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:30.111871 2186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:30.111949 2186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:30.112021 2186 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 04:39:30.112066 2186 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:39:30.112133 2186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:39:30.117301 2186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.166.190:45297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:30.118373 2186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:30.119436 2186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:30.129060 2194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:30.129158 2186 server_base.cc:1034] running on GCE node
W20260806 04:39:30.129038 2197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:30.129063 2195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:30.129717 2186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:30.130306 2186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:30.131489 2186 hybrid_clock.cc:648] HybridClock initialized: now 1785991170131476 us; error 23 us; skew 500 ppm
Aug 06 04:39:30 dist-test-slave-vkjp krb5kdc[1753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991170, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:30.134310 2186 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:39:30.135236 2186 webserver.cc:460] Webserver started at http://127.31.166.131:43321/ using document root <none> and password file <none>
I20260806 04:39:30.135780 2186 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:30.135879 2186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:30.136106 2186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:30.138074 2186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "545d98e4ed6a4ef48b74366c61acfd01"
format_stamp: "Formatted at 2026-08-06 04:39:30 on dist-test-slave-vkjp"
server_key: "68c02c305159c6ee0a38b037e2fe2ff7"
server_key_iv: "fb19f552e9e18a6131d15bd2fcac54c3"
server_key_version: "encryptionkey@0"
I20260806 04:39:30.138628 2186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "545d98e4ed6a4ef48b74366c61acfd01"
format_stamp: "Formatted at 2026-08-06 04:39:30 on dist-test-slave-vkjp"
server_key: "68c02c305159c6ee0a38b037e2fe2ff7"
server_key_iv: "fb19f552e9e18a6131d15bd2fcac54c3"
server_key_version: "encryptionkey@0"
I20260806 04:39:30.142339 2186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260806 04:39:30.144729 2204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:30.146006 2186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:39:30.146143 2186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "545d98e4ed6a4ef48b74366c61acfd01"
format_stamp: "Formatted at 2026-08-06 04:39:30 on dist-test-slave-vkjp"
server_key: "68c02c305159c6ee0a38b037e2fe2ff7"
server_key_iv: "fb19f552e9e18a6131d15bd2fcac54c3"
server_key_version: "encryptionkey@0"
I20260806 04:39:30.146260 2186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:30.157707 2186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:30.160699 2186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:30.160908 2186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:30.162467 2186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:30.162536 2186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:30.162609 2186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:30.162663 2186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:39:30 dist-test-slave-vkjp krb5kdc[1753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991170, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:30.173241 2186 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:42429
I20260806 04:39:30.173274 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:42429 every 8 connection(s)
I20260806 04:39:30.174091 2186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:39:30.176761 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.163225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:47221 (local address 127.31.166.190:45297)
0806 04:39:30.163398 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.163403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.164629 (+ 1226us) server_negotiation.cc:407] Connection header received
0806 04:39:30.165708 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.165712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.165772 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.165878 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:30.167013 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.167933 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.169226 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.169486 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.169607 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.173195 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:30.173199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:30.173204 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:30.173236 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:30.175003 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:30.175482 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:30.175485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:30.175486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:30.175561 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:30.175867 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:30.175869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:30.175870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:30.176035 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:30.176108 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.176542 (+ 434us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.176650 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 04:39:30.177544 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.164034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:45297 (local address 127.31.166.131:47221)
0806 04:39:30.164465 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:30.164513 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:30.165509 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:30.166046 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:30.166052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:30.166366 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:30.166797 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.166810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.168093 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.168098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.169102 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.169111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.169654 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.169659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.169871 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.171313 (+ 1442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:30.171334 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:30.173001 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:30.175132 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:30.175138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:30.175156 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:30.175394 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:30.175644 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:30.175646 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:30.175648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:30.175766 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:30.176120 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:30.176125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:30.176327 (+ 202us) client_negotiation.cc:769] Sending connection context
0806 04:39:30.176468 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 04:39:30.176699 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":131,"threads_started":1}
I20260806 04:39:30.178838 2325 heartbeater.cc:347] Connected to a master server at 127.31.166.190:45297
I20260806 04:39:30.178997 2325 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:30.179487 2325 heartbeater.cc:511] Master 127.31.166.190:45297 requested a full tablet report, sending...
I20260806 04:39:30.180474 1803 ts_manager.cc:194] Registered new tserver with Master: 545d98e4ed6a4ef48b74366c61acfd01 (127.31.166.131:42429)
I20260806 04:39:30.180670 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 2186
I20260806 04:39:30.180773 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:39:30.181083 32410 external_mini_cluster.cc:1428] Setting key 42ea061a7b73ecc420129a1dc8d405dd
I20260806 04:39:30.181243 1803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:47221
I20260806 04:39:30.183007 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:39:30.189640 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.180964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:45297 (local address 127.31.166.131:37097)
0806 04:39:30.181270 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:30.181281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:30.181570 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:30.183075 (+ 1505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:30.183077 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:30.183293 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:30.183387 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.183393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.184116 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.184119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.184693 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.184704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.185222 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.185226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.185333 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.185796 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:30.185810 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:30.186206 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:30.188033 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:30.188039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:30.188043 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:30.188299 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:30.188652 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:30.188655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:30.188656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:30.188739 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:30.189225 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:30.189227 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:30.189298 (+ 71us) client_negotiation.cc:769] Sending connection context
0806 04:39:30.189385 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 04:39:30.189492 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":82,"threads_started":1}
I20260806 04:39:30.189844 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.181027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:37097 (local address 127.31.166.190:45297)
0806 04:39:30.181136 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.181140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.181352 (+ 212us) server_negotiation.cc:407] Connection header received
0806 04:39:30.181657 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.181660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.181698 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.181793 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:30.183499 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.183994 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.184881 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.185108 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.185187 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.186390 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:30.186393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:30.186395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:30.186416 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:30.187925 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:30.188432 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:30.188437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:30.188439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:30.188534 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:30.188840 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:30.188867 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:30.188872 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:30.189127 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:30.189235 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.189475 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.189610 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Aug 06 04:39:30 dist-test-slave-vkjp krb5kdc[1753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 04:39:30 dist-test-slave-vkjp krb5kdc[1753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:30.210837 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.202055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.31.166.190:45297)
0806 04:39:30.202183 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.202186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.202198 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:39:30.202234 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.202236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.202273 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.202365 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:30.203047 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.203717 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.204710 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.205127 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.205255 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.207990 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:30.207993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:30.207995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:30.208017 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:30.209249 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:30.209647 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:30.209650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:30.209651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:30.209722 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:30.209961 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:30.209963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:30.209964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:30.210120 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:30.210216 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.210564 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.210655 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:39:30.213965 1803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:39:30.216954 1803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:39:30.232344 2344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.225382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:45595 (local address 127.0.0.1:37554)
0806 04:39:30.226380 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:30.226431 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:30.226559 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:30.227514 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:30.227521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:30.227545 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:30.227694 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.227702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.229215 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.229219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.230669 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.230681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.231881 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.231884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.232009 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.232033 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 04:39:30.232110 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 04:39:30.232190 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":28,"thread_start_us":73,"threads_started":1}
I20260806 04:39:30.232780 2349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.227156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.31.166.130:44441)
0806 04:39:30.227672 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.227682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.227702 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:39:30.227804 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.227810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.227996 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.228155 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:30.228514 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.229509 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.230628 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.231495 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.231647 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.231798 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.232540 (+ 742us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.232638 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":135,"threads_started":1}
I20260806 04:39:30.232890 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.226324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37800 (local address 127.31.166.131:42429)
0806 04:39:30.227334 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.227371 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.227744 (+ 373us) server_negotiation.cc:407] Connection header received
0806 04:39:30.227846 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.227849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.227982 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.228125 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:30.228347 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.229416 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.230922 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.231695 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.231859 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.232020 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.232624 (+ 604us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.232727 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":139,"threads_started":1}
I20260806 04:39:30.233064 2345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.225483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.31.166.129:45595)
0806 04:39:30.225849 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.225865 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.226832 (+ 967us) server_negotiation.cc:407] Connection header received
0806 04:39:30.226918 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.226923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.227160 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.227327 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:30.227827 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.229061 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.230809 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.231607 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.231732 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.231847 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.232698 (+ 851us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.232807 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":77,"threads_started":1}
I20260806 04:39:30.232352 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.225590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:44441 (local address 127.0.0.1:50646)
0806 04:39:30.226859 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:30.226880 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:30.227508 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:30.228269 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:30.228274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:30.228294 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:30.228400 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.228404 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.229632 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.229638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.230502 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.230511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.231634 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.231639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.231792 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.231848 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 04:39:30.231975 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 04:39:30.232105 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"mutex_wait_us":44,"thread_start_us":125,"threads_started":1}
I20260806 04:39:30.233161 2346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.225858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:42429 (local address 127.0.0.1:37800)
0806 04:39:30.227550 (+ 1692us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:30.227564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:30.227645 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:30.228117 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:30.228120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:30.228137 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:30.228228 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.228237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.229575 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.229578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.230776 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.230786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.232096 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.232100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.232200 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.232225 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:39:30.232420 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 04:39:30.232509 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1608,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260806 04:39:30.236040 1959 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5a8baeaafa428ca0624b068636e6ac (DEFAULT_TABLE table=test-table [id=d533c47b578640e79912c628105231da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:30.236043 2108 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5a8baeaafa428ca0624b068636e6ac (DEFAULT_TABLE table=test-table [id=d533c47b578640e79912c628105231da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:30.236043 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5a8baeaafa428ca0624b068636e6ac (DEFAULT_TABLE table=test-table [id=d533c47b578640e79912c628105231da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:30.237000 1959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5a8baeaafa428ca0624b068636e6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:30.237020 2108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5a8baeaafa428ca0624b068636e6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:30.237030 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5a8baeaafa428ca0624b068636e6ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:30.244675 2350 tablet_bootstrap.cc:492] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840: Bootstrap starting.
I20260806 04:39:30.244936 2351 tablet_bootstrap.cc:492] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01: Bootstrap starting.
I20260806 04:39:30.245275 2352 tablet_bootstrap.cc:492] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a: Bootstrap starting.
I20260806 04:39:30.246608 2350 tablet_bootstrap.cc:654] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:30.246990 2352 tablet_bootstrap.cc:654] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:30.247476 2350 log.cc:824] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:30.247723 2352 log.cc:824] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:30.249459 2350 tablet_bootstrap.cc:492] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840: No bootstrap required, opened a new log
I20260806 04:39:30.249634 2352 tablet_bootstrap.cc:492] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a: No bootstrap required, opened a new log
I20260806 04:39:30.249722 2350 ts_tablet_manager.cc:1397] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 04:39:30.249881 2352 ts_tablet_manager.cc:1397] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:39:30.249990 2351 tablet_bootstrap.cc:654] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:30.250684 2351 log.cc:824] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:30.252269 2351 tablet_bootstrap.cc:492] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01: No bootstrap required, opened a new log
I20260806 04:39:30.252472 2351 ts_tablet_manager.cc:1397] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260806 04:39:30.253731 2352 raft_consensus.cc:348] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.253731 2350 raft_consensus.cc:348] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.253993 2352 raft_consensus.cc:374] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:30.253993 2350 raft_consensus.cc:374] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:30.254052 2350 raft_consensus.cc:729] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964cb62944bc40daadba7dadcf2ad840, State: Initialized, Role: FOLLOWER
I20260806 04:39:30.254079 2352 raft_consensus.cc:729] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6815f42fd2384e43aa7f714b43e3753a, State: Initialized, Role: FOLLOWER
I20260806 04:39:30.254556 2352 consensus_queue.cc:260] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.254556 2350 consensus_queue.cc:260] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.255517 2177 heartbeater.cc:503] Master 127.31.166.190:45297 was elected leader, sending a full tablet report...
I20260806 04:39:30.255580 2352 ts_tablet_manager.cc:1428] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260806 04:39:30.255580 2350 ts_tablet_manager.cc:1428] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260806 04:39:30.255885 2027 heartbeater.cc:503] Master 127.31.166.190:45297 was elected leader, sending a full tablet report...
I20260806 04:39:30.257622 2351 raft_consensus.cc:348] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.257900 2351 raft_consensus.cc:374] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:30.258009 2351 raft_consensus.cc:729] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545d98e4ed6a4ef48b74366c61acfd01, State: Initialized, Role: FOLLOWER
I20260806 04:39:30.258451 2351 consensus_queue.cc:260] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.259573 2325 heartbeater.cc:503] Master 127.31.166.190:45297 was elected leader, sending a full tablet report...
I20260806 04:39:30.259917 2351 ts_tablet_manager.cc:1428] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:39:30.273315 2358 raft_consensus.cc:482] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:39:30.273478 2358 raft_consensus.cc:504] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.274438 2358 leader_election.cc:288] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 964cb62944bc40daadba7dadcf2ad840 (127.31.166.129:45595), 6815f42fd2384e43aa7f714b43e3753a (127.31.166.130:44441)
I20260806 04:39:30.278477 2345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.274775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:34455 (local address 127.31.166.129:45595)
0806 04:39:30.275020 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.275025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.275042 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:39:30.275126 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.275132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.275188 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.275287 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:30.275629 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.276245 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.277520 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.277967 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.278085 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.278138 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.278275 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.278346 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260806 04:39:30.278523 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.274687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:45595 (local address 127.31.166.131:34455)
0806 04:39:30.274903 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:30.274918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:30.275009 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:30.275343 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:30.275346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:30.275365 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:30.275475 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.275480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.276359 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.276362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.277394 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.277404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.278076 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.278079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.278162 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.278183 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 04:39:30.278276 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 04:39:30.278361 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 04:39:30.279215 2349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.274775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:59173 (local address 127.31.166.130:44441)
0806 04:39:30.274902 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.274907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.275270 (+ 363us) server_negotiation.cc:407] Connection header received
0806 04:39:30.275422 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.275426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.275471 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.275522 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:30.276215 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.276760 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.277923 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.278506 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.278647 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.278706 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.279025 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.279115 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:39:30.279225 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.274687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:44441 (local address 127.31.166.131:59173)
0806 04:39:30.275162 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:30.275178 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:30.275275 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:30.275695 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:30.275698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:30.275716 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:30.276096 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.276103 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.276872 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.276876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.277812 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.277819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.278785 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.278790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.278892 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.278919 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 04:39:30.278998 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 04:39:30.279066 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":40,"thread_start_us":119,"threads_started":1}
I20260806 04:39:30.279292 1979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5a8baeaafa428ca0624b068636e6ac" candidate_uuid: "545d98e4ed6a4ef48b74366c61acfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964cb62944bc40daadba7dadcf2ad840" is_pre_election: true
I20260806 04:39:30.279636 1979 raft_consensus.cc:2454] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 545d98e4ed6a4ef48b74366c61acfd01 in term 0.
I20260806 04:39:30.279798 2128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5a8baeaafa428ca0624b068636e6ac" candidate_uuid: "545d98e4ed6a4ef48b74366c61acfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6815f42fd2384e43aa7f714b43e3753a" is_pre_election: true
I20260806 04:39:30.280077 2128 raft_consensus.cc:2454] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 545d98e4ed6a4ef48b74366c61acfd01 in term 0.
I20260806 04:39:30.280171 2205 leader_election.cc:302] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 545d98e4ed6a4ef48b74366c61acfd01, 964cb62944bc40daadba7dadcf2ad840; no voters:
I20260806 04:39:30.280444 2358 raft_consensus.cc:2781] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:39:30.280543 2358 raft_consensus.cc:482] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:39:30.280578 2358 raft_consensus.cc:3037] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:30.281630 2358 raft_consensus.cc:504] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.281982 2358 leader_election.cc:288] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [CANDIDATE]: Term 1 election: Requested vote from peers 964cb62944bc40daadba7dadcf2ad840 (127.31.166.129:45595), 6815f42fd2384e43aa7f714b43e3753a (127.31.166.130:44441)
I20260806 04:39:30.282367 2128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5a8baeaafa428ca0624b068636e6ac" candidate_uuid: "545d98e4ed6a4ef48b74366c61acfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6815f42fd2384e43aa7f714b43e3753a"
I20260806 04:39:30.282387 1979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5a8baeaafa428ca0624b068636e6ac" candidate_uuid: "545d98e4ed6a4ef48b74366c61acfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964cb62944bc40daadba7dadcf2ad840"
I20260806 04:39:30.282469 2128 raft_consensus.cc:3037] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:30.282514 1979 raft_consensus.cc:3037] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:30.283591 2128 raft_consensus.cc:2454] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 545d98e4ed6a4ef48b74366c61acfd01 in term 1.
I20260806 04:39:30.283588 1979 raft_consensus.cc:2454] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 545d98e4ed6a4ef48b74366c61acfd01 in term 1.
I20260806 04:39:30.283902 2208 leader_election.cc:302] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 545d98e4ed6a4ef48b74366c61acfd01, 6815f42fd2384e43aa7f714b43e3753a; no voters:
I20260806 04:39:30.284093 2358 raft_consensus.cc:2781] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:30.284332 2358 raft_consensus.cc:686] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 LEADER]: Becoming Leader. State: Replica: 545d98e4ed6a4ef48b74366c61acfd01, State: Running, Role: LEADER
I20260806 04:39:30.284641 2358 consensus_queue.cc:237] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.287758 1802 catalog_manager.cc:5495] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 545d98e4ed6a4ef48b74366c61acfd01 (127.31.166.131). New cstate: current_term: 1 leader_uuid: "545d98e4ed6a4ef48b74366c61acfd01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } health_report { overall_health: HEALTHY } } }
Aug 06 04:39:30 dist-test-slave-vkjp krb5kdc[1753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991169, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:30.333488 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.324787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:45297 (local address 127.0.0.1:53198)
0806 04:39:30.325166 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:30.325185 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:30.325371 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:30.325632 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:30.325634 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:30.326015 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:30.326115 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.326122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.326894 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.326897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.327578 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.327588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.328010 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.328013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.328081 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.328674 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:30.328687 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:30.330159 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:30.331935 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:30.331943 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:30.331948 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:30.332174 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:30.332481 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:30.332483 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:30.332485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:30.332564 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:30.332918 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:30.332923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:30.333031 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 04:39:30.333183 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 04:39:30.333323 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":143,"threads_started":1}
I20260806 04:39:30.333508 1861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.324946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53198 (local address 127.31.166.190:45297)
0806 04:39:30.325066 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.325071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.325307 (+ 236us) server_negotiation.cc:407] Connection header received
0806 04:39:30.325474 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.325477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.325517 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.325586 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:30.326236 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.326773 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.327696 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.327902 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.327980 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.330288 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:30.330291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:30.330293 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:30.330334 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:30.331803 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:30.332308 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:30.332310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:30.332312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:30.332386 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:30.332644 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:30.332646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:30.332647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:30.332831 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:30.332907 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.333246 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.333385 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":49}
I20260806 04:39:30.337553 1802 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:53198:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 04:39:30.338258 1802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:39:30.343146 1959 tablet_service.cc:1467] Processing CreateTablet for tablet 1a924150a04b461c9cb39c3d2dcc0c36 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be374521f9634582b9c92eec25d3771a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:39:30.343292 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 1a924150a04b461c9cb39c3d2dcc0c36 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be374521f9634582b9c92eec25d3771a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:39:30.343329 2108 tablet_service.cc:1467] Processing CreateTablet for tablet 1a924150a04b461c9cb39c3d2dcc0c36 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be374521f9634582b9c92eec25d3771a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:39:30.343878 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a924150a04b461c9cb39c3d2dcc0c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:30.343883 1959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a924150a04b461c9cb39c3d2dcc0c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:30.343971 2108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a924150a04b461c9cb39c3d2dcc0c36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:30.346310 2352 tablet_bootstrap.cc:492] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a: Bootstrap starting.
I20260806 04:39:30.346700 2351 tablet_bootstrap.cc:492] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01: Bootstrap starting.
I20260806 04:39:30.346817 2350 tablet_bootstrap.cc:492] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840: Bootstrap starting.
I20260806 04:39:30.347481 2352 tablet_bootstrap.cc:654] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:30.347743 2351 tablet_bootstrap.cc:654] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:30.347743 2350 tablet_bootstrap.cc:654] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:30.349071 2352 tablet_bootstrap.cc:492] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a: No bootstrap required, opened a new log
I20260806 04:39:30.349156 2350 tablet_bootstrap.cc:492] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840: No bootstrap required, opened a new log
I20260806 04:39:30.349184 2352 ts_tablet_manager.cc:1397] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:39:30.349251 2350 ts_tablet_manager.cc:1397] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:39:30.349344 2351 tablet_bootstrap.cc:492] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01: No bootstrap required, opened a new log
I20260806 04:39:30.349438 2351 ts_tablet_manager.cc:1397] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 04:39:30.349715 2352 raft_consensus.cc:348] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.349731 2350 raft_consensus.cc:348] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.349839 2352 raft_consensus.cc:374] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:30.349839 2350 raft_consensus.cc:374] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:30.349884 2350 raft_consensus.cc:729] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964cb62944bc40daadba7dadcf2ad840, State: Initialized, Role: FOLLOWER
I20260806 04:39:30.349884 2352 raft_consensus.cc:729] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6815f42fd2384e43aa7f714b43e3753a, State: Initialized, Role: FOLLOWER
I20260806 04:39:30.349850 2351 raft_consensus.cc:348] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.349957 2351 raft_consensus.cc:374] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:30.350001 2351 raft_consensus.cc:729] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545d98e4ed6a4ef48b74366c61acfd01, State: Initialized, Role: FOLLOWER
I20260806 04:39:30.350009 2350 consensus_queue.cc:260] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.350015 2352 consensus_queue.cc:260] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.350119 2351 consensus_queue.cc:260] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.350298 2352 ts_tablet_manager.cc:1428] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:39:30.350303 2350 ts_tablet_manager.cc:1428] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:39:30.350404 2358 tablet_replica.cc:439] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } } }
I20260806 04:39:30.350525 2358 tablet_replica.cc:442] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:39:30.350476 2357 tablet_replica.cc:439] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } } }
I20260806 04:39:30.350541 2356 tablet_replica.cc:439] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } } }
I20260806 04:39:30.350597 2357 tablet_replica.cc:442] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:39:30.350651 2356 tablet_replica.cc:442] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:39:30.350759 2351 ts_tablet_manager.cc:1428] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260806 04:39:30.425048 2326 tablet.cc:2369] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:39:30.451901 2358 raft_consensus.cc:482] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:39:30.452039 2358 raft_consensus.cc:504] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.452406 2358 leader_election.cc:288] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 964cb62944bc40daadba7dadcf2ad840 (127.31.166.129:45595), 6815f42fd2384e43aa7f714b43e3753a (127.31.166.130:44441)
I20260806 04:39:30.452925 2128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a924150a04b461c9cb39c3d2dcc0c36" candidate_uuid: "545d98e4ed6a4ef48b74366c61acfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6815f42fd2384e43aa7f714b43e3753a" is_pre_election: true
I20260806 04:39:30.452930 1979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a924150a04b461c9cb39c3d2dcc0c36" candidate_uuid: "545d98e4ed6a4ef48b74366c61acfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964cb62944bc40daadba7dadcf2ad840" is_pre_election: true
I20260806 04:39:30.453071 1979 raft_consensus.cc:2454] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 545d98e4ed6a4ef48b74366c61acfd01 in term 0.
I20260806 04:39:30.453095 2128 raft_consensus.cc:2454] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 545d98e4ed6a4ef48b74366c61acfd01 in term 0.
I20260806 04:39:30.453462 2208 leader_election.cc:302] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 545d98e4ed6a4ef48b74366c61acfd01, 6815f42fd2384e43aa7f714b43e3753a; no voters:
I20260806 04:39:30.453660 2358 raft_consensus.cc:2781] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:39:30.453724 2358 raft_consensus.cc:482] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:39:30.453799 2358 raft_consensus.cc:3037] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:30.454954 2358 raft_consensus.cc:504] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.455416 2358 leader_election.cc:288] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [CANDIDATE]: Term 1 election: Requested vote from peers 964cb62944bc40daadba7dadcf2ad840 (127.31.166.129:45595), 6815f42fd2384e43aa7f714b43e3753a (127.31.166.130:44441)
I20260806 04:39:30.455797 2128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a924150a04b461c9cb39c3d2dcc0c36" candidate_uuid: "545d98e4ed6a4ef48b74366c61acfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6815f42fd2384e43aa7f714b43e3753a"
I20260806 04:39:30.455950 2128 raft_consensus.cc:3037] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:30.457158 2128 raft_consensus.cc:2454] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 545d98e4ed6a4ef48b74366c61acfd01 in term 1.
I20260806 04:39:30.457451 1979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a924150a04b461c9cb39c3d2dcc0c36" candidate_uuid: "545d98e4ed6a4ef48b74366c61acfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964cb62944bc40daadba7dadcf2ad840"
I20260806 04:39:30.457504 2208 leader_election.cc:302] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 545d98e4ed6a4ef48b74366c61acfd01, 6815f42fd2384e43aa7f714b43e3753a; no voters:
I20260806 04:39:30.457590 1979 raft_consensus.cc:3037] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:30.457671 2358 raft_consensus.cc:2781] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:30.457747 2358 raft_consensus.cc:686] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 LEADER]: Becoming Leader. State: Replica: 545d98e4ed6a4ef48b74366c61acfd01, State: Running, Role: LEADER
I20260806 04:39:30.457906 2358 consensus_queue.cc:237] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
W20260806 04:39:30.458659 2178 tablet.cc:2369] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:39:30.458889 1979 raft_consensus.cc:2454] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 545d98e4ed6a4ef48b74366c61acfd01 in term 1.
I20260806 04:39:30.459390 2360 tablet_replica.cc:439] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01: TxnStatusTablet state changed. Reason: New leader 545d98e4ed6a4ef48b74366c61acfd01. Latest consensus state: current_term: 1 leader_uuid: "545d98e4ed6a4ef48b74366c61acfd01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } } }
I20260806 04:39:30.459509 2360 tablet_replica.cc:442] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:39:30.459828 2373 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 04:39:30.460003 2373 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 04:39:30.460911 1802 catalog_manager.cc:5495] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 545d98e4ed6a4ef48b74366c61acfd01 (127.31.166.131). New cstate: current_term: 1 leader_uuid: "545d98e4ed6a4ef48b74366c61acfd01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } health_report { overall_health: HEALTHY } } }
I20260806 04:39:30.477880 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.473662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.31.166.131:42429)
0806 04:39:30.473793 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.473796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.473846 (+ 50us) server_negotiation.cc:407] Connection header received
0806 04:39:30.474058 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.474060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.474112 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.474194 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:30.474570 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.475147 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.476488 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.476697 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.476776 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.477121 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:30.477503 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:30.477586 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.477709 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.477760 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:39:30.477952 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.473515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:42429 (local address 127.0.0.1:35398)
0806 04:39:30.473767 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:30.473785 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:30.473918 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:30.474281 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:30.474288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:30.474304 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:39:30.474436 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.474443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.475273 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.475277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.476357 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.476370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.476815 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.476819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.476926 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.476981 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:39:30.477597 (+ 616us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:39:30.477612 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 04:39:30.477691 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 04:39:30.477782 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260806 04:39:30.483832 1979 raft_consensus.cc:1261] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Refusing update from remote peer 545d98e4ed6a4ef48b74366c61acfd01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:30.483834 2128 raft_consensus.cc:1261] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Refusing update from remote peer 545d98e4ed6a4ef48b74366c61acfd01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:30.484624 2360 consensus_queue.cc:1035] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:30.485194 2360 consensus_queue.cc:1035] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:30.488950 2356 tablet_replica.cc:439] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840: TxnStatusTablet state changed. Reason: New leader 545d98e4ed6a4ef48b74366c61acfd01. Latest consensus state: current_term: 1 leader_uuid: "545d98e4ed6a4ef48b74366c61acfd01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } } }
I20260806 04:39:30.489108 2356 tablet_replica.cc:442] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:39:30.490576 2357 tablet_replica.cc:439] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a: TxnStatusTablet state changed. Reason: New leader 545d98e4ed6a4ef48b74366c61acfd01. Latest consensus state: current_term: 1 leader_uuid: "545d98e4ed6a4ef48b74366c61acfd01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } } }
I20260806 04:39:30.490708 2357 tablet_replica.cc:442] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:39:30.493496 2376 mvcc.cc:205] Tried to move back new op lower bound from 7315419834294292480 to 7315419834196934656. Current Snapshot: MvccSnapshot[applied={T|T < 7315419834294292480}]
I20260806 04:39:30.493839 2358 tablet_replica.cc:439] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "545d98e4ed6a4ef48b74366c61acfd01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } } }
I20260806 04:39:30.493963 2358 tablet_replica.cc:442] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:39:30.494104 2358 tablet_replica.cc:439] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "545d98e4ed6a4ef48b74366c61acfd01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } } }
I20260806 04:39:30.494199 2358 tablet_replica.cc:442] T 1a924150a04b461c9cb39c3d2dcc0c36 P 545d98e4ed6a4ef48b74366c61acfd01: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:39:30.494613 2356 tablet_replica.cc:439] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "545d98e4ed6a4ef48b74366c61acfd01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } } }
I20260806 04:39:30.494728 2356 tablet_replica.cc:442] T 1a924150a04b461c9cb39c3d2dcc0c36 P 964cb62944bc40daadba7dadcf2ad840: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:39:30.495289 2377 mvcc.cc:205] Tried to move back new op lower bound from 7315419834294292480 to 7315419834196934656. Current Snapshot: MvccSnapshot[applied={T|T < 7315419834294292480}]
W20260806 04:39:30.495676 2028 tablet.cc:2369] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:39:30.496089 2357 tablet_replica.cc:439] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "545d98e4ed6a4ef48b74366c61acfd01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } } }
I20260806 04:39:30.496202 2357 tablet_replica.cc:442] T 1a924150a04b461c9cb39c3d2dcc0c36 P 6815f42fd2384e43aa7f714b43e3753a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:39:30.497383 2378 mvcc.cc:205] Tried to move back new op lower bound from 7315419834294292480 to 7315419834196934656. Current Snapshot: MvccSnapshot[applied={T|T < 7315419834294292480}]
I20260806 04:39:30.505216 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.501233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35406 (local address 127.31.166.131:42429)
0806 04:39:30.501341 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.501345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.501358 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:39:30.501400 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.501403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.501452 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.501530 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:30.501823 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.502600 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.503927 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.504327 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.504433 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.504786 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:30.504907 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:30.504970 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.505067 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.505115 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 04:39:30.516414 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.512882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:37915 (local address 127.31.166.131:42429)
0806 04:39:30.513001 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.513005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.513094 (+ 89us) server_negotiation.cc:407] Connection header received
0806 04:39:30.513214 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.513217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.513259 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.513331 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:30.513589 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.514120 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.515346 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.515566 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.515660 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.515968 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:30.516040 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:30.516098 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.516208 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.516265 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":45}
I20260806 04:39:30.516456 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.512797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:42429 (local address 127.31.166.131:37915)
0806 04:39:30.513019 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:30.513033 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:30.513125 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:30.513362 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:30.513365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:30.513372 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:39:30.513486 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.513492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.514226 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.514229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.515212 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.515222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.515671 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.515675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.515766 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.515815 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:39:30.516123 (+ 308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:39:30.516130 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:39:30.516202 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 04:39:30.516305 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":19}
I20260806 04:39:30.523607 2128 raft_consensus.cc:1261] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Refusing update from remote peer 545d98e4ed6a4ef48b74366c61acfd01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:30.523743 1979 raft_consensus.cc:1261] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Refusing update from remote peer 545d98e4ed6a4ef48b74366c61acfd01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:30.524369 2384 consensus_queue.cc:1035] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:30.524549 2360 consensus_queue.cc:1035] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:30.571177 1803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:30.571349 1803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:30.573815 1803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06: Sending DeleteTablet for 3 replicas of tablet 4a5a8baeaafa428ca0624b068636e6ac
I20260806 04:39:30.574569 1959 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a5a8baeaafa428ca0624b068636e6ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:30 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37554
I20260806 04:39:30.574728 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a5a8baeaafa428ca0624b068636e6ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:30 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37800
I20260806 04:39:30.575139 2108 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a5a8baeaafa428ca0624b068636e6ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:30 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:50646
I20260806 04:39:30.575268 2393 tablet_replica.cc:330] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840: stopping tablet replica
I20260806 04:39:30.575542 1803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 04:39:30.575649 2393 raft_consensus.cc:2229] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:30.575866 2393 raft_consensus.cc:2258] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 04:39:30.575975 1803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:39:30.575992 2395 tablet_replica.cc:330] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a: stopping tablet replica
I20260806 04:39:30.576647 2395 raft_consensus.cc:2229] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:30.576866 2394 tablet_replica.cc:330] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01: stopping tablet replica
I20260806 04:39:30.576967 2395 raft_consensus.cc:2258] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:30.577122 2394 raft_consensus.cc:2229] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:39:30.577491 2394 raft_consensus.cc:2258] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:30.578435 2393 ts_tablet_manager.cc:1905] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:30.578570 2395 ts_tablet_manager.cc:1905] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:30.579980 2394 ts_tablet_manager.cc:1905] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:30.582113 2395 ts_tablet_manager.cc:1918] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:39:30.582264 2395 log.cc:1196] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/4a5a8baeaafa428ca0624b068636e6ac
I20260806 04:39:30.582664 2395 ts_tablet_manager.cc:1939] T 4a5a8baeaafa428ca0624b068636e6ac P 6815f42fd2384e43aa7f714b43e3753a: Deleting consensus metadata
I20260806 04:39:30.582927 2394 ts_tablet_manager.cc:1918] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:39:30.583031 2394 log.cc:1196] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/4a5a8baeaafa428ca0624b068636e6ac
I20260806 04:39:30.583375 2394 ts_tablet_manager.cc:1939] T 4a5a8baeaafa428ca0624b068636e6ac P 545d98e4ed6a4ef48b74366c61acfd01: Deleting consensus metadata
I20260806 04:39:30.583377 1959 tablet_service.cc:1467] Processing CreateTablet for tablet daa50a491fa44f5790659c923d500d7e (DEFAULT_TABLE table=test-table [id=594bb88f8bc64fb4b4472138ce989859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:30.583653 1959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa50a491fa44f5790659c923d500d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:30.583750 2258 tablet_service.cc:1467] Processing CreateTablet for tablet daa50a491fa44f5790659c923d500d7e (DEFAULT_TABLE table=test-table [id=594bb88f8bc64fb4b4472138ce989859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:30.583998 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa50a491fa44f5790659c923d500d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:30.584517 1787 catalog_manager.cc:4841] TS 545d98e4ed6a4ef48b74366c61acfd01 (127.31.166.131:42429): tablet 4a5a8baeaafa428ca0624b068636e6ac (table test-table [id=d533c47b578640e79912c628105231da]) successfully deleted
I20260806 04:39:30.586277 2108 tablet_service.cc:1467] Processing CreateTablet for tablet daa50a491fa44f5790659c923d500d7e (DEFAULT_TABLE table=test-table [id=594bb88f8bc64fb4b4472138ce989859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:30.586436 2351 tablet_bootstrap.cc:492] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01: Bootstrap starting.
I20260806 04:39:30.586833 2350 tablet_bootstrap.cc:492] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840: Bootstrap starting.
I20260806 04:39:30.587353 2351 tablet_bootstrap.cc:654] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:30.587395 2108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa50a491fa44f5790659c923d500d7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:30.587384 2393 ts_tablet_manager.cc:1918] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:39:30.587469 2393 log.cc:1196] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/wal/wals/4a5a8baeaafa428ca0624b068636e6ac
I20260806 04:39:30.587795 2393 ts_tablet_manager.cc:1939] T 4a5a8baeaafa428ca0624b068636e6ac P 964cb62944bc40daadba7dadcf2ad840: Deleting consensus metadata
I20260806 04:39:30.589017 2351 tablet_bootstrap.cc:492] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01: No bootstrap required, opened a new log
I20260806 04:39:30.589095 2351 ts_tablet_manager.cc:1397] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:39:30.589460 2351 raft_consensus.cc:348] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.589551 2351 raft_consensus.cc:374] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:30.589572 2351 raft_consensus.cc:729] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545d98e4ed6a4ef48b74366c61acfd01, State: Initialized, Role: FOLLOWER
I20260806 04:39:30.589709 2351 consensus_queue.cc:260] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.589870 2352 tablet_bootstrap.cc:492] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a: Bootstrap starting.
I20260806 04:39:30.590003 2351 ts_tablet_manager.cc:1428] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:39:30.590770 2352 tablet_bootstrap.cc:654] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:30.591760 1785 catalog_manager.cc:4841] TS 964cb62944bc40daadba7dadcf2ad840 (127.31.166.129:45595): tablet 4a5a8baeaafa428ca0624b068636e6ac (table test-table [id=d533c47b578640e79912c628105231da]) successfully deleted
I20260806 04:39:30.592285 2350 tablet_bootstrap.cc:654] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:30.593186 1789 catalog_manager.cc:4841] TS 6815f42fd2384e43aa7f714b43e3753a (127.31.166.130:44441): tablet 4a5a8baeaafa428ca0624b068636e6ac (table test-table [id=d533c47b578640e79912c628105231da]) successfully deleted
I20260806 04:39:30.593468 2350 tablet_bootstrap.cc:492] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840: No bootstrap required, opened a new log
I20260806 04:39:30.593559 2350 ts_tablet_manager.cc:1397] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260806 04:39:30.593806 2352 tablet_bootstrap.cc:492] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a: No bootstrap required, opened a new log
I20260806 04:39:30.593916 2352 ts_tablet_manager.cc:1397] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260806 04:39:30.593986 2350 raft_consensus.cc:348] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.594116 2350 raft_consensus.cc:374] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:30.594154 2350 raft_consensus.cc:729] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964cb62944bc40daadba7dadcf2ad840, State: Initialized, Role: FOLLOWER
I20260806 04:39:30.594283 2350 consensus_queue.cc:260] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.594563 2350 ts_tablet_manager.cc:1428] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:39:30.594475 2352 raft_consensus.cc:348] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.594622 2352 raft_consensus.cc:374] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:30.594661 2352 raft_consensus.cc:729] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6815f42fd2384e43aa7f714b43e3753a, State: Initialized, Role: FOLLOWER
I20260806 04:39:30.594797 2352 consensus_queue.cc:260] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.595084 2352 ts_tablet_manager.cc:1428] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260806 04:39:30.673961 2356 raft_consensus.cc:482] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:39:30.674134 2356 raft_consensus.cc:504] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.675454 2356 leader_election.cc:288] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6815f42fd2384e43aa7f714b43e3753a (127.31.166.130:44441), 545d98e4ed6a4ef48b74366c61acfd01 (127.31.166.131:42429)
I20260806 04:39:30.680787 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.675896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:35975 (local address 127.31.166.131:42429)
0806 04:39:30.676026 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.676030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.676518 (+ 488us) server_negotiation.cc:407] Connection header received
0806 04:39:30.676672 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.676675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.676725 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.676816 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:30.677115 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.677714 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.679276 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.680037 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.680154 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.680209 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.680577 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.680653 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 04:39:30.681106 2349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.676146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:39901 (local address 127.31.166.130:44441)
0806 04:39:30.676277 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.676282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.676556 (+ 274us) server_negotiation.cc:407] Connection header received
0806 04:39:30.676953 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.676957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.677001 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.677069 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:30.677380 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.678100 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.679584 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.680274 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.680416 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.680484 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.680871 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.680953 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 04:39:30.681384 2400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.675715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:42429 (local address 127.31.166.129:35975)
0806 04:39:30.676410 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:30.676427 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:30.676564 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:30.676862 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:30.676864 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:30.676882 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:30.676990 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.676995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.677824 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.677839 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.679064 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.679106 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.680306 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.680310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.680433 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.680457 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 04:39:30.681070 (+ 613us) client_negotiation.cc:240] Negotiation successful
0806 04:39:30.681148 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":231,"spinlock_wait_cycles":22272,"thread_start_us":129,"threads_started":1}
I20260806 04:39:30.681725 2399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.675715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:44441 (local address 127.31.166.129:39901)
0806 04:39:30.676410 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:30.676427 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:30.676564 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:30.677136 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:30.677138 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:30.677154 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:30.677266 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.677273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.678221 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.678224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.679461 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.679470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.680564 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.680570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.680711 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.680735 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 04:39:30.681303 (+ 568us) client_negotiation.cc:240] Negotiation successful
0806 04:39:30.681371 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"spinlock_wait_cycles":14720,"thread_start_us":75,"threads_started":1}
I20260806 04:39:30.681990 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa50a491fa44f5790659c923d500d7e" candidate_uuid: "964cb62944bc40daadba7dadcf2ad840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "545d98e4ed6a4ef48b74366c61acfd01" is_pre_election: true
I20260806 04:39:30.682185 2278 raft_consensus.cc:2454] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 964cb62944bc40daadba7dadcf2ad840 in term 0.
I20260806 04:39:30.682591 1910 leader_election.cc:302] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 545d98e4ed6a4ef48b74366c61acfd01, 964cb62944bc40daadba7dadcf2ad840; no voters:
I20260806 04:39:30.682680 2128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa50a491fa44f5790659c923d500d7e" candidate_uuid: "964cb62944bc40daadba7dadcf2ad840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6815f42fd2384e43aa7f714b43e3753a" is_pre_election: true
I20260806 04:39:30.682826 2128 raft_consensus.cc:2454] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 964cb62944bc40daadba7dadcf2ad840 in term 0.
I20260806 04:39:30.682870 2356 raft_consensus.cc:2781] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:39:30.682957 2356 raft_consensus.cc:482] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:39:30.683010 2356 raft_consensus.cc:3037] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:30.684259 2356 raft_consensus.cc:504] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.684687 2356 leader_election.cc:288] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [CANDIDATE]: Term 1 election: Requested vote from peers 6815f42fd2384e43aa7f714b43e3753a (127.31.166.130:44441), 545d98e4ed6a4ef48b74366c61acfd01 (127.31.166.131:42429)
I20260806 04:39:30.684963 2128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa50a491fa44f5790659c923d500d7e" candidate_uuid: "964cb62944bc40daadba7dadcf2ad840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6815f42fd2384e43aa7f714b43e3753a"
I20260806 04:39:30.685083 2128 raft_consensus.cc:3037] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:30.686094 2128 raft_consensus.cc:2454] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 964cb62944bc40daadba7dadcf2ad840 in term 1.
I20260806 04:39:30.686359 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daa50a491fa44f5790659c923d500d7e" candidate_uuid: "964cb62944bc40daadba7dadcf2ad840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "545d98e4ed6a4ef48b74366c61acfd01"
I20260806 04:39:30.686408 1911 leader_election.cc:302] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6815f42fd2384e43aa7f714b43e3753a, 964cb62944bc40daadba7dadcf2ad840; no voters:
I20260806 04:39:30.686470 2278 raft_consensus.cc:3037] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:30.686587 2356 raft_consensus.cc:2781] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:30.686856 2356 raft_consensus.cc:686] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [term 1 LEADER]: Becoming Leader. State: Replica: 964cb62944bc40daadba7dadcf2ad840, State: Running, Role: LEADER
I20260806 04:39:30.687139 2356 consensus_queue.cc:237] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:30.687718 2278 raft_consensus.cc:2454] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 964cb62944bc40daadba7dadcf2ad840 in term 1.
I20260806 04:39:30.689646 1802 catalog_manager.cc:5495] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 reported cstate change: term changed from 0 to 1, leader changed from <none> to 964cb62944bc40daadba7dadcf2ad840 (127.31.166.129). New cstate: current_term: 1 leader_uuid: "964cb62944bc40daadba7dadcf2ad840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } health_report { overall_health: UNKNOWN } } }
I20260806 04:39:30.722745 2345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.719109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.31.166.129:45595)
0806 04:39:30.719233 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.719239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.719255 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:39:30.719477 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.719480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.719546 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.719657 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:30.719915 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.720444 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.721274 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.721515 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.721624 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.721893 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:30.722359 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:30.722487 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.722546 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.722598 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:39:30.734498 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.730248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:38937 (local address 127.31.166.131:42429)
0806 04:39:30.730414 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.730419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.730540 (+ 121us) server_negotiation.cc:407] Connection header received
0806 04:39:30.730698 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.730701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.730755 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.730837 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:30.731121 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.731626 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.733058 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.733561 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.733711 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.734026 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:30.734099 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:30.734163 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.734325 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.734389 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 04:39:30.734555 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.730133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:42429 (local address 127.31.166.129:38937)
0806 04:39:30.730450 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:30.730468 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:30.730596 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:30.730859 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:30.730862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:30.730869 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:39:30.730988 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.730993 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.731742 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.731748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.732910 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.732919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.733715 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.733718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.733815 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.733868 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:39:30.734194 (+ 326us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:39:30.734202 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 04:39:30.734276 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 04:39:30.734375 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":111,"threads_started":1}
I20260806 04:39:30.740974 2128 raft_consensus.cc:1261] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Refusing update from remote peer 964cb62944bc40daadba7dadcf2ad840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:30.741310 2278 raft_consensus.cc:1261] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Refusing update from remote peer 964cb62944bc40daadba7dadcf2ad840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:30.741497 2356 consensus_queue.cc:1035] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:30.741822 2401 consensus_queue.cc:1035] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:30.765431 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.761250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:45595 (local address 127.31.166.131:52931)
0806 04:39:30.761504 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:30.761522 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:30.761597 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:30.762203 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:30.762208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:30.762218 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:39:30.762319 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.762325 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.763287 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.763292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.764009 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:30.764022 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.764666 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.764672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:30.764753 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.764783 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:39:30.765119 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:39:30.765126 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:39:30.765189 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 04:39:30.765278 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260806 04:39:30.765882 2345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:30.761736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:52931 (local address 127.31.166.129:45595)
0806 04:39:30.761880 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:30.761885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:30.761898 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:39:30.761940 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:30.761942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:30.761993 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:30.762068 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:30.762460 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.763176 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.764154 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:30.764500 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:30.764594 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:30.764918 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:30.765006 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:30.765079 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:30.765727 (+ 648us) server_negotiation.cc:299] Negotiation successful
0806 04:39:30.765780 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 04:39:30.766489 2377 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 04:39:32.330053 1802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:32.330252 1802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:32.333097 1802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06: Sending DeleteTablet for 3 replicas of tablet daa50a491fa44f5790659c923d500d7e
I20260806 04:39:32.333884 2108 tablet_service.cc:1514] Processing DeleteTablet for tablet daa50a491fa44f5790659c923d500d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:32 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:50646
I20260806 04:39:32.333909 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet daa50a491fa44f5790659c923d500d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:32 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37800
I20260806 04:39:32.334223 1959 tablet_service.cc:1514] Processing DeleteTablet for tablet daa50a491fa44f5790659c923d500d7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:32 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37554
I20260806 04:39:32.334424 2434 tablet_replica.cc:330] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a: stopping tablet replica
I20260806 04:39:32.334568 2434 raft_consensus.cc:2229] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:32.334687 2434 raft_consensus.cc:2258] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:32.335461 2434 ts_tablet_manager.cc:1905] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:32.336594 2435 tablet_replica.cc:330] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01: stopping tablet replica
I20260806 04:39:32.336845 2435 raft_consensus.cc:2229] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:32.337078 2435 raft_consensus.cc:2258] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:32.337211 2436 tablet_replica.cc:330] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840: stopping tablet replica
I20260806 04:39:32.337385 2436 raft_consensus.cc:2229] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:39:32.337689 2436 raft_consensus.cc:2258] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:32.337872 2435 ts_tablet_manager.cc:1905] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:32.338546 2434 ts_tablet_manager.cc:1918] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:39:32.338634 2434 log.cc:1196] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/daa50a491fa44f5790659c923d500d7e
I20260806 04:39:32.338934 2434 ts_tablet_manager.cc:1939] T daa50a491fa44f5790659c923d500d7e P 6815f42fd2384e43aa7f714b43e3753a: Deleting consensus metadata
I20260806 04:39:32.339529 2436 ts_tablet_manager.cc:1905] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:32.339780 1789 catalog_manager.cc:4841] TS 6815f42fd2384e43aa7f714b43e3753a (127.31.166.130:44441): tablet daa50a491fa44f5790659c923d500d7e (table test-table [id=594bb88f8bc64fb4b4472138ce989859]) successfully deleted
I20260806 04:39:32.341427 2435 ts_tablet_manager.cc:1918] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:39:32.341523 2435 log.cc:1196] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/daa50a491fa44f5790659c923d500d7e
I20260806 04:39:32.341863 2435 ts_tablet_manager.cc:1939] T daa50a491fa44f5790659c923d500d7e P 545d98e4ed6a4ef48b74366c61acfd01: Deleting consensus metadata
I20260806 04:39:32.342347 2436 ts_tablet_manager.cc:1918] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:39:32.342439 2436 log.cc:1196] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/wal/wals/daa50a491fa44f5790659c923d500d7e
I20260806 04:39:32.342654 1787 catalog_manager.cc:4841] TS 545d98e4ed6a4ef48b74366c61acfd01 (127.31.166.131:42429): tablet daa50a491fa44f5790659c923d500d7e (table test-table [id=594bb88f8bc64fb4b4472138ce989859]) successfully deleted
I20260806 04:39:32.342777 2436 ts_tablet_manager.cc:1939] T daa50a491fa44f5790659c923d500d7e P 964cb62944bc40daadba7dadcf2ad840: Deleting consensus metadata
I20260806 04:39:32.343590 1785 catalog_manager.cc:4841] TS 964cb62944bc40daadba7dadcf2ad840 (127.31.166.129:45595): tablet daa50a491fa44f5790659c923d500d7e (table test-table [id=594bb88f8bc64fb4b4472138ce989859]) successfully deleted
Aug 06 04:39:32 dist-test-slave-vkjp krb5kdc[1753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 04:39:32 dist-test-slave-vkjp krb5kdc[1753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991172, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:32.365509 2444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:32.355948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54692 (local address 127.31.166.190:45297)
0806 04:39:32.356273 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:32.356280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:32.356296 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:39:32.356480 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:32.356484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:32.356535 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:32.356684 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:32.357598 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.358148 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.359019 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.359345 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.359431 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:32.361895 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:32.361899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:32.361901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:32.361925 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:32.363653 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:32.364209 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:32.364212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:32.364213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:32.364301 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:32.364626 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:32.364628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:32.364629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:32.364809 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:32.364919 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:32.365125 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 04:39:32.365234 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":121,"threads_started":1}
I20260806 04:39:32.368130 1802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:39:32.368533 1802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:39:32.373174 1959 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb2211fee1e48b0be024e4523bdf34b (DEFAULT_TABLE table=test-table [id=4e237b0a35fe4080a4a756a258a07731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:32.373340 2108 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb2211fee1e48b0be024e4523bdf34b (DEFAULT_TABLE table=test-table [id=4e237b0a35fe4080a4a756a258a07731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:32.373418 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 7bb2211fee1e48b0be024e4523bdf34b (DEFAULT_TABLE table=test-table [id=4e237b0a35fe4080a4a756a258a07731]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:32.373524 1959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb2211fee1e48b0be024e4523bdf34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:32.373641 2108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb2211fee1e48b0be024e4523bdf34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:32.373711 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bb2211fee1e48b0be024e4523bdf34b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:32.375749 2445 tablet_bootstrap.cc:492] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a: Bootstrap starting.
I20260806 04:39:32.376269 2446 tablet_bootstrap.cc:492] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01: Bootstrap starting.
I20260806 04:39:32.376732 2447 tablet_bootstrap.cc:492] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840: Bootstrap starting.
I20260806 04:39:32.377283 2445 tablet_bootstrap.cc:654] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:32.377283 2446 tablet_bootstrap.cc:654] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:32.377566 2447 tablet_bootstrap.cc:654] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:32.378635 2445 tablet_bootstrap.cc:492] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a: No bootstrap required, opened a new log
I20260806 04:39:32.378780 2445 ts_tablet_manager.cc:1397] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:39:32.378840 2446 tablet_bootstrap.cc:492] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01: No bootstrap required, opened a new log
I20260806 04:39:32.378952 2446 ts_tablet_manager.cc:1397] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:39:32.379112 2447 tablet_bootstrap.cc:492] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840: No bootstrap required, opened a new log
I20260806 04:39:32.379235 2447 ts_tablet_manager.cc:1397] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:39:32.379388 2445 raft_consensus.cc:348] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.379433 2446 raft_consensus.cc:348] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.379519 2445 raft_consensus.cc:374] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:32.379572 2446 raft_consensus.cc:374] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:32.379587 2445 raft_consensus.cc:729] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6815f42fd2384e43aa7f714b43e3753a, State: Initialized, Role: FOLLOWER
I20260806 04:39:32.379608 2446 raft_consensus.cc:729] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545d98e4ed6a4ef48b74366c61acfd01, State: Initialized, Role: FOLLOWER
I20260806 04:39:32.379724 2446 consensus_queue.cc:260] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.379760 2445 consensus_queue.cc:260] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.379781 2447 raft_consensus.cc:348] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.379880 2447 raft_consensus.cc:374] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:32.379920 2447 raft_consensus.cc:729] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964cb62944bc40daadba7dadcf2ad840, State: Initialized, Role: FOLLOWER
I20260806 04:39:32.380038 2446 ts_tablet_manager.cc:1428] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260806 04:39:32.380043 2447 consensus_queue.cc:260] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.380302 2447 ts_tablet_manager.cc:1428] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 04:39:32.380630 2445 ts_tablet_manager.cc:1428] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:39:32.485725 2360 raft_consensus.cc:482] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:39:32.485882 2360 raft_consensus.cc:504] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.486377 2360 leader_election.cc:288] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 964cb62944bc40daadba7dadcf2ad840 (127.31.166.129:45595), 6815f42fd2384e43aa7f714b43e3753a (127.31.166.130:44441)
I20260806 04:39:32.486822 2128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb2211fee1e48b0be024e4523bdf34b" candidate_uuid: "545d98e4ed6a4ef48b74366c61acfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6815f42fd2384e43aa7f714b43e3753a" is_pre_election: true
I20260806 04:39:32.486838 1979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb2211fee1e48b0be024e4523bdf34b" candidate_uuid: "545d98e4ed6a4ef48b74366c61acfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964cb62944bc40daadba7dadcf2ad840" is_pre_election: true
I20260806 04:39:32.486996 2128 raft_consensus.cc:2454] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 545d98e4ed6a4ef48b74366c61acfd01 in term 0.
I20260806 04:39:32.487013 1979 raft_consensus.cc:2454] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 545d98e4ed6a4ef48b74366c61acfd01 in term 0.
I20260806 04:39:32.487373 2205 leader_election.cc:302] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 545d98e4ed6a4ef48b74366c61acfd01, 964cb62944bc40daadba7dadcf2ad840; no voters:
I20260806 04:39:32.487593 2360 raft_consensus.cc:2781] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:39:32.487646 2360 raft_consensus.cc:482] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:39:32.487694 2360 raft_consensus.cc:3037] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:32.488715 2360 raft_consensus.cc:504] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.489108 2360 leader_election.cc:288] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [CANDIDATE]: Term 1 election: Requested vote from peers 964cb62944bc40daadba7dadcf2ad840 (127.31.166.129:45595), 6815f42fd2384e43aa7f714b43e3753a (127.31.166.130:44441)
I20260806 04:39:32.489457 2128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb2211fee1e48b0be024e4523bdf34b" candidate_uuid: "545d98e4ed6a4ef48b74366c61acfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6815f42fd2384e43aa7f714b43e3753a"
I20260806 04:39:32.489461 1979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bb2211fee1e48b0be024e4523bdf34b" candidate_uuid: "545d98e4ed6a4ef48b74366c61acfd01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964cb62944bc40daadba7dadcf2ad840"
I20260806 04:39:32.489590 1979 raft_consensus.cc:3037] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:32.489596 2128 raft_consensus.cc:3037] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:32.490490 2128 raft_consensus.cc:2454] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 545d98e4ed6a4ef48b74366c61acfd01 in term 1.
I20260806 04:39:32.490718 1979 raft_consensus.cc:2454] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 545d98e4ed6a4ef48b74366c61acfd01 in term 1.
I20260806 04:39:32.490796 2208 leader_election.cc:302] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 545d98e4ed6a4ef48b74366c61acfd01, 6815f42fd2384e43aa7f714b43e3753a; no voters:
I20260806 04:39:32.490976 2360 raft_consensus.cc:2781] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:32.491117 2360 raft_consensus.cc:686] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 LEADER]: Becoming Leader. State: Replica: 545d98e4ed6a4ef48b74366c61acfd01, State: Running, Role: LEADER
I20260806 04:39:32.491269 2360 consensus_queue.cc:237] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.492985 1803 catalog_manager.cc:5495] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 545d98e4ed6a4ef48b74366c61acfd01 (127.31.166.131). New cstate: current_term: 1 leader_uuid: "545d98e4ed6a4ef48b74366c61acfd01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } health_report { overall_health: HEALTHY } } }
I20260806 04:39:32.511547 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:32.508194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35420 (local address 127.31.166.131:42429)
0806 04:39:32.508507 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:32.508515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:32.508529 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:39:32.508570 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:32.508572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:32.508622 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:32.508701 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:32.508963 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.509564 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.510384 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.510608 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.510692 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:32.510966 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:32.511035 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:32.511178 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:32.511308 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 04:39:32.511376 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":166,"threads_started":1}
I20260806 04:39:32.517918 2128 raft_consensus.cc:1261] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Refusing update from remote peer 545d98e4ed6a4ef48b74366c61acfd01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:32.517917 1979 raft_consensus.cc:1261] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Refusing update from remote peer 545d98e4ed6a4ef48b74366c61acfd01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:32.518314 2453 consensus_queue.cc:1035] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:32.518527 2360 consensus_queue.cc:1035] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:32.553506 1803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:32.553658 1803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:32.555907 1803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06: Sending DeleteTablet for 3 replicas of tablet 7bb2211fee1e48b0be024e4523bdf34b
I20260806 04:39:32.556569 1959 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bb2211fee1e48b0be024e4523bdf34b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:32 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37554
I20260806 04:39:32.556574 2108 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bb2211fee1e48b0be024e4523bdf34b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:32 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:50646
I20260806 04:39:32.556809 2436 tablet_replica.cc:330] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840: stopping tablet replica
I20260806 04:39:32.556818 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bb2211fee1e48b0be024e4523bdf34b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:32 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37800
I20260806 04:39:32.556968 2436 raft_consensus.cc:2229] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:32.557018 2435 tablet_replica.cc:330] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01: stopping tablet replica
I20260806 04:39:32.557123 2436 raft_consensus.cc:2258] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:32.557130 2435 raft_consensus.cc:2229] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:39:32.557310 2435 raft_consensus.cc:2258] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:32.557312 2434 tablet_replica.cc:330] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a: stopping tablet replica
I20260806 04:39:32.557443 2434 raft_consensus.cc:2229] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:32.557559 2434 raft_consensus.cc:2258] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:32.557955 2436 ts_tablet_manager.cc:1905] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:32.558162 2435 ts_tablet_manager.cc:1905] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:32.558564 2434 ts_tablet_manager.cc:1905] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:32.558486 1803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:39:32.558898 1803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:39:32.561026 2436 ts_tablet_manager.cc:1918] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:39:32.561074 2435 ts_tablet_manager.cc:1918] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:39:32.561074 2434 ts_tablet_manager.cc:1918] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:39:32.561111 2436 log.cc:1196] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-0/wal/wals/7bb2211fee1e48b0be024e4523bdf34b
I20260806 04:39:32.561146 2434 log.cc:1196] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/7bb2211fee1e48b0be024e4523bdf34b
I20260806 04:39:32.561146 2435 log.cc:1196] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/7bb2211fee1e48b0be024e4523bdf34b
I20260806 04:39:32.561434 2435 ts_tablet_manager.cc:1939] T 7bb2211fee1e48b0be024e4523bdf34b P 545d98e4ed6a4ef48b74366c61acfd01: Deleting consensus metadata
I20260806 04:39:32.561457 2434 ts_tablet_manager.cc:1939] T 7bb2211fee1e48b0be024e4523bdf34b P 6815f42fd2384e43aa7f714b43e3753a: Deleting consensus metadata
I20260806 04:39:32.561827 2436 ts_tablet_manager.cc:1939] T 7bb2211fee1e48b0be024e4523bdf34b P 964cb62944bc40daadba7dadcf2ad840: Deleting consensus metadata
I20260806 04:39:32.562139 1789 catalog_manager.cc:4841] TS 6815f42fd2384e43aa7f714b43e3753a (127.31.166.130:44441): tablet 7bb2211fee1e48b0be024e4523bdf34b (table test-table [id=4e237b0a35fe4080a4a756a258a07731]) successfully deleted
I20260806 04:39:32.562764 1787 catalog_manager.cc:4841] TS 545d98e4ed6a4ef48b74366c61acfd01 (127.31.166.131:42429): tablet 7bb2211fee1e48b0be024e4523bdf34b (table test-table [id=4e237b0a35fe4080a4a756a258a07731]) successfully deleted
I20260806 04:39:32.562816 1785 catalog_manager.cc:4841] TS 964cb62944bc40daadba7dadcf2ad840 (127.31.166.129:45595): tablet 7bb2211fee1e48b0be024e4523bdf34b (table test-table [id=4e237b0a35fe4080a4a756a258a07731]) successfully deleted
I20260806 04:39:32.564638 1959 tablet_service.cc:1467] Processing CreateTablet for tablet c9bde12459cd4330935a609f66861f47 (DEFAULT_TABLE table=test-table [id=43768150fd4b44d8ac7f41d6352a50a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:32.564824 2258 tablet_service.cc:1467] Processing CreateTablet for tablet c9bde12459cd4330935a609f66861f47 (DEFAULT_TABLE table=test-table [id=43768150fd4b44d8ac7f41d6352a50a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:32.564975 1959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9bde12459cd4330935a609f66861f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:32.564900 2108 tablet_service.cc:1467] Processing CreateTablet for tablet c9bde12459cd4330935a609f66861f47 (DEFAULT_TABLE table=test-table [id=43768150fd4b44d8ac7f41d6352a50a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:32.565095 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9bde12459cd4330935a609f66861f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:32.565220 2108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9bde12459cd4330935a609f66861f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:32.567121 2445 tablet_bootstrap.cc:492] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a: Bootstrap starting.
I20260806 04:39:32.567504 2446 tablet_bootstrap.cc:492] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01: Bootstrap starting.
I20260806 04:39:32.568158 2445 tablet_bootstrap.cc:654] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:32.568295 2446 tablet_bootstrap.cc:654] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:32.568778 2447 tablet_bootstrap.cc:492] T c9bde12459cd4330935a609f66861f47 P 964cb62944bc40daadba7dadcf2ad840: Bootstrap starting.
I20260806 04:39:32.569412 2445 tablet_bootstrap.cc:492] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a: No bootstrap required, opened a new log
I20260806 04:39:32.569473 2447 tablet_bootstrap.cc:654] T c9bde12459cd4330935a609f66861f47 P 964cb62944bc40daadba7dadcf2ad840: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:32.569531 2445 ts_tablet_manager.cc:1397] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:39:32.570058 2445 raft_consensus.cc:348] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.570186 2445 raft_consensus.cc:374] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:32.570235 2445 raft_consensus.cc:729] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6815f42fd2384e43aa7f714b43e3753a, State: Initialized, Role: FOLLOWER
I20260806 04:39:32.570310 2446 tablet_bootstrap.cc:492] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01: No bootstrap required, opened a new log
I20260806 04:39:32.570420 2446 ts_tablet_manager.cc:1397] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260806 04:39:32.570396 2445 consensus_queue.cc:260] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.570717 2445 ts_tablet_manager.cc:1428] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:39:32.570859 2446 raft_consensus.cc:348] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.570957 2446 raft_consensus.cc:374] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:32.570979 2446 raft_consensus.cc:729] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 545d98e4ed6a4ef48b74366c61acfd01, State: Initialized, Role: FOLLOWER
I20260806 04:39:32.571132 2446 consensus_queue.cc:260] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.571424 2446 ts_tablet_manager.cc:1428] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260806 04:39:32.571614 2447 tablet_bootstrap.cc:492] T c9bde12459cd4330935a609f66861f47 P 964cb62944bc40daadba7dadcf2ad840: No bootstrap required, opened a new log
I20260806 04:39:32.571724 2447 ts_tablet_manager.cc:1397] T c9bde12459cd4330935a609f66861f47 P 964cb62944bc40daadba7dadcf2ad840: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 04:39:32.572252 2447 raft_consensus.cc:348] T c9bde12459cd4330935a609f66861f47 P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.572395 2447 raft_consensus.cc:374] T c9bde12459cd4330935a609f66861f47 P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:32.572433 2447 raft_consensus.cc:729] T c9bde12459cd4330935a609f66861f47 P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964cb62944bc40daadba7dadcf2ad840, State: Initialized, Role: FOLLOWER
I20260806 04:39:32.572568 2447 consensus_queue.cc:260] T c9bde12459cd4330935a609f66861f47 P 964cb62944bc40daadba7dadcf2ad840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.572883 2447 ts_tablet_manager.cc:1428] T c9bde12459cd4330935a609f66861f47 P 964cb62944bc40daadba7dadcf2ad840: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 04:39:32.628868 2451 raft_consensus.cc:482] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:39:32.629078 2451 raft_consensus.cc:504] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.630456 2451 leader_election.cc:288] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 964cb62944bc40daadba7dadcf2ad840 (127.31.166.129:45595), 545d98e4ed6a4ef48b74366c61acfd01 (127.31.166.131:42429)
I20260806 04:39:32.636168 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:32.630845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:42941 (local address 127.31.166.129:45595)
0806 04:39:32.631151 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:32.631163 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:32.631192 (+ 29us) server_negotiation.cc:407] Connection header received
0806 04:39:32.631413 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:32.631416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:32.631463 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:32.631547 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:32.632003 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.632826 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.634719 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.635415 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.635522 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:32.635565 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:32.635940 (+ 375us) server_negotiation.cc:299] Negotiation successful
0806 04:39:32.636017 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
I20260806 04:39:32.636154 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:32.630906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:34477 (local address 127.31.166.131:42429)
0806 04:39:32.631038 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:32.631043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:32.631296 (+ 253us) server_negotiation.cc:407] Connection header received
0806 04:39:32.631386 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:32.631389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:32.631449 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:32.631543 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:32.632003 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.632822 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.634719 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.635372 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.635510 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:32.635566 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:32.635940 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 04:39:32.636017 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 04:39:32.636214 2462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:32.630696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:45595 (local address 127.31.166.130:42941)
0806 04:39:32.631087 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:32.631129 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:32.631252 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:32.631571 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:32.631574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:32.631614 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:32.631879 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:32.631887 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.632997 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.632999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:32.634583 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:32.634595 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.635535 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.635540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:32.635744 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:32.635812 (+ 68us) client_negotiation.cc:769] Sending connection context
0806 04:39:32.635908 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 04:39:32.636018 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":314,"spinlock_wait_cycles":38144,"thread_start_us":99,"threads_started":1}
I20260806 04:39:32.636212 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:32.630696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:42429 (local address 127.31.166.130:34477)
0806 04:39:32.631206 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:32.631219 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:32.631305 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:32.631571 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:32.631574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:32.631615 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:32.631883 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:32.631888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.632980 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.632985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:32.634589 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:32.634595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.635540 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.635543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:32.635741 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:32.635841 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 04:39:32.635908 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 04:39:32.636018 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"mutex_wait_us":342,"spinlock_wait_cycles":4096,"thread_start_us":118,"threads_started":1}
I20260806 04:39:32.636768 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9bde12459cd4330935a609f66861f47" candidate_uuid: "6815f42fd2384e43aa7f714b43e3753a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "545d98e4ed6a4ef48b74366c61acfd01" is_pre_election: true
I20260806 04:39:32.636991 2278 raft_consensus.cc:2454] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6815f42fd2384e43aa7f714b43e3753a in term 0.
I20260806 04:39:32.637192 1979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9bde12459cd4330935a609f66861f47" candidate_uuid: "6815f42fd2384e43aa7f714b43e3753a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964cb62944bc40daadba7dadcf2ad840" is_pre_election: true
I20260806 04:39:32.637346 1979 raft_consensus.cc:2454] T c9bde12459cd4330935a609f66861f47 P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6815f42fd2384e43aa7f714b43e3753a in term 0.
I20260806 04:39:32.637478 2058 leader_election.cc:302] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 545d98e4ed6a4ef48b74366c61acfd01, 6815f42fd2384e43aa7f714b43e3753a; no voters:
I20260806 04:39:32.637756 2451 raft_consensus.cc:2781] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:39:32.637818 2451 raft_consensus.cc:482] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:39:32.637871 2451 raft_consensus.cc:3037] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:32.638904 2451 raft_consensus.cc:504] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.639256 2451 leader_election.cc:288] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [CANDIDATE]: Term 1 election: Requested vote from peers 964cb62944bc40daadba7dadcf2ad840 (127.31.166.129:45595), 545d98e4ed6a4ef48b74366c61acfd01 (127.31.166.131:42429)
I20260806 04:39:32.639557 1979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9bde12459cd4330935a609f66861f47" candidate_uuid: "6815f42fd2384e43aa7f714b43e3753a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "964cb62944bc40daadba7dadcf2ad840"
I20260806 04:39:32.639612 2278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9bde12459cd4330935a609f66861f47" candidate_uuid: "6815f42fd2384e43aa7f714b43e3753a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "545d98e4ed6a4ef48b74366c61acfd01"
I20260806 04:39:32.639729 1979 raft_consensus.cc:3037] T c9bde12459cd4330935a609f66861f47 P 964cb62944bc40daadba7dadcf2ad840 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:32.639744 2278 raft_consensus.cc:3037] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:32.640658 2278 raft_consensus.cc:2454] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6815f42fd2384e43aa7f714b43e3753a in term 1.
I20260806 04:39:32.640667 1979 raft_consensus.cc:2454] T c9bde12459cd4330935a609f66861f47 P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6815f42fd2384e43aa7f714b43e3753a in term 1.
I20260806 04:39:32.640950 2056 leader_election.cc:302] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6815f42fd2384e43aa7f714b43e3753a, 964cb62944bc40daadba7dadcf2ad840; no voters:
I20260806 04:39:32.641158 2451 raft_consensus.cc:2781] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:32.641399 2451 raft_consensus.cc:686] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [term 1 LEADER]: Becoming Leader. State: Replica: 6815f42fd2384e43aa7f714b43e3753a, State: Running, Role: LEADER
I20260806 04:39:32.641649 2451 consensus_queue.cc:237] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } }
I20260806 04:39:32.644028 1801 catalog_manager.cc:5495] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6815f42fd2384e43aa7f714b43e3753a (127.31.166.130). New cstate: current_term: 1 leader_uuid: "6815f42fd2384e43aa7f714b43e3753a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6815f42fd2384e43aa7f714b43e3753a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 44441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 } health_report { overall_health: UNKNOWN } } }
I20260806 04:39:32.679744 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:32.674980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.31.166.130:44441)
0806 04:39:32.675343 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:32.675353 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:32.675366 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:39:32.675453 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:32.675455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:32.675509 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:32.675653 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:32.675928 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.676990 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.678022 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.678298 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.678403 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:32.678715 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:32.679304 (+ 589us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:32.679411 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:32.679484 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 04:39:32.679540 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":172,"threads_started":1}
I20260806 04:39:32.690385 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:32.687364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:43985 (local address 127.31.166.131:42429)
0806 04:39:32.687511 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:32.687516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:32.687714 (+ 198us) server_negotiation.cc:407] Connection header received
0806 04:39:32.687805 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:32.687807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:32.687858 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:32.687921 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:32.688148 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.688660 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.689483 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.689684 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.689758 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:32.690015 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:32.690081 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:32.690135 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:32.690220 (+ 85us) server_negotiation.cc:299] Negotiation successful
0806 04:39:32.690265 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:39:32.690436 2470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:32.687270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:42429 (local address 127.31.166.130:43985)
0806 04:39:32.687633 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:32.687645 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:32.687734 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:32.687947 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:32.687949 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:32.687955 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:39:32.688054 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:32.688058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.688754 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.688756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:32.689383 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:32.689391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.689780 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.689782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:32.689864 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:32.689905 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:39:32.690158 (+ 253us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:39:32.690163 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:39:32.690212 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 04:39:32.690277 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":134,"threads_started":1}
I20260806 04:39:32.696794 1979 raft_consensus.cc:1261] T c9bde12459cd4330935a609f66861f47 P 964cb62944bc40daadba7dadcf2ad840 [term 1 FOLLOWER]: Refusing update from remote peer 6815f42fd2384e43aa7f714b43e3753a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:32.696784 2278 raft_consensus.cc:1261] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Refusing update from remote peer 6815f42fd2384e43aa7f714b43e3753a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:32.697289 2465 consensus_queue.cc:1035] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [LEADER]: Connected to new peer: Peer: permanent_uuid: "964cb62944bc40daadba7dadcf2ad840" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:32.697479 2451 consensus_queue.cc:1035] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [LEADER]: Connected to new peer: Peer: permanent_uuid: "545d98e4ed6a4ef48b74366c61acfd01" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:32.714986 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:32.711512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:56313 (local address 127.31.166.130:44441)
0806 04:39:32.711644 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:32.711649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:32.712038 (+ 389us) server_negotiation.cc:407] Connection header received
0806 04:39:32.712095 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:32.712097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:32.712150 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:32.712227 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:32.712410 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.712907 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.713815 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.714174 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.714299 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:32.714570 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:32.714644 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:32.714735 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:32.714823 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 04:39:32.714882 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:39:32.714993 2475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:32.711460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:44441 (local address 127.31.166.131:56313)
0806 04:39:32.711782 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:32.711794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:32.711939 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:32.712245 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:32.712247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:32.712254 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:39:32.712343 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:32.712348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.713008 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.713011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:32.713694 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:32.713703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:32.714288 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:32.714293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:32.714395 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:32.714430 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:39:32.714747 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:39:32.714753 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 04:39:32.714809 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 04:39:32.714882 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":114,"threads_started":1}
W20260806 04:39:32.715634 2415 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 04:39:34.273787 1801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:34.274008 1801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:34.276631 1801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f20edfa9e07c40579859dd5d8cebea06: Sending DeleteTablet for 3 replicas of tablet c9bde12459cd4330935a609f66861f47
I20260806 04:39:34.277266 1959 tablet_service.cc:1514] Processing DeleteTablet for tablet c9bde12459cd4330935a609f66861f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:34 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37554
I20260806 04:39:34.277442 2108 tablet_service.cc:1514] Processing DeleteTablet for tablet c9bde12459cd4330935a609f66861f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:34 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:50646
I20260806 04:39:34.277446 2258 tablet_service.cc:1514] Processing DeleteTablet for tablet c9bde12459cd4330935a609f66861f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:34 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37800
I20260806 04:39:34.277794 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 1886
I20260806 04:39:34.277809 2501 tablet_replica.cc:330] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a: stopping tablet replica
I20260806 04:39:34.277801 2502 tablet_replica.cc:330] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01: stopping tablet replica
I20260806 04:39:34.279997 2502 raft_consensus.cc:2229] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:34.280158 2502 raft_consensus.cc:2258] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:34.281028 2502 ts_tablet_manager.cc:1905] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:34.281236 2501 raft_consensus.cc:2229] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:39:34.281622 2501 raft_consensus.cc:2258] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:34.282263 2501 ts_tablet_manager.cc:1905] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:34.284152 2502 ts_tablet_manager.cc:1918] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:39:34.284245 2502 log.cc:1196] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/c9bde12459cd4330935a609f66861f47
I20260806 04:39:34.284564 2502 ts_tablet_manager.cc:1939] T c9bde12459cd4330935a609f66861f47 P 545d98e4ed6a4ef48b74366c61acfd01: Deleting consensus metadata
I20260806 04:39:34.285334 1787 catalog_manager.cc:4841] TS 545d98e4ed6a4ef48b74366c61acfd01 (127.31.166.131:42429): tablet c9bde12459cd4330935a609f66861f47 (table test-table [id=43768150fd4b44d8ac7f41d6352a50a1]) successfully deleted
I20260806 04:39:34.285379 2501 ts_tablet_manager.cc:1918] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 04:39:34.285462 2501 log.cc:1196] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/c9bde12459cd4330935a609f66861f47
I20260806 04:39:34.285765 2501 ts_tablet_manager.cc:1939] T c9bde12459cd4330935a609f66861f47 P 6815f42fd2384e43aa7f714b43e3753a: Deleting consensus metadata
I20260806 04:39:34.286442 1789 catalog_manager.cc:4841] TS 6815f42fd2384e43aa7f714b43e3753a (127.31.166.130:44441): tablet c9bde12459cd4330935a609f66861f47 (table test-table [id=43768150fd4b44d8ac7f41d6352a50a1]) successfully deleted
W20260806 04:39:34.289779 1785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:45595 (error 108)
I20260806 04:39:34.290045 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 2037
I20260806 04:39:34.292003 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:34.291499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:45595 (local address 127.0.0.1:38564)
0806 04:39:34.291772 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:34.291873 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:45595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":134,"threads_started":1}
W20260806 04:39:34.293097 1785 catalog_manager.cc:4584] TS 964cb62944bc40daadba7dadcf2ad840 (127.31.166.129:45595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c9bde12459cd4330935a609f66861f47: Network error: Client connection negotiation failed: client connection to 127.31.166.129:45595: connect: Connection refused (error 111)
I20260806 04:39:34.299786 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 2186
I20260806 04:39:34.307848 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 1768
2026-08-06T04:39:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6206 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:39:34 dist-test-slave-vkjp krb5kdc[2507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:39:34 dist-test-slave-vkjp krb5kdc[2507](info): set up 2 sockets
Aug 06 04:39:34 dist-test-slave-vkjp krb5kdc[2507](info): commencing operation
krb5kdc: starting...
W20260806 04:39:35.391871 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:39:35 dist-test-slave-vkjp krb5kdc[2507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:39:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:39:35.519220 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:35253
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:35253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:35.642596 2522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:35.642963 2522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:35.643117 2522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:35.647612 2522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:35.647714 2522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:35.647753 2522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:35.647833 2522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:35.647871 2522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:39:35.653631 2522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:35253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:35253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:35.654844 2522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:35.655975 2522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:35.664714 2529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:35.664739 2530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:35.664893 2532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:35.665277 2522 server_base.cc:1034] running on GCE node
I20260806 04:39:35.665853 2522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:35.667021 2522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:35.668239 2522 hybrid_clock.cc:648] HybridClock initialized: now 1785991175668223 us; error 28 us; skew 500 ppm
Aug 06 04:39:35 dist-test-slave-vkjp krb5kdc[2507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991175, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:35.671059 2522 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:39:35.672240 2522 webserver.cc:460] Webserver started at http://127.31.166.190:35453/ using document root <none> and password file <none>
I20260806 04:39:35.673098 2522 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:35.673205 2522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:35.673453 2522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:35.675686 2522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "86f24b7ccdd944e9ba5ca37395a45b15"
format_stamp: "Formatted at 2026-08-06 04:39:35 on dist-test-slave-vkjp"
server_key: "ebc70b9516332af2d9c394ad93297b89"
server_key_iv: "c5a5c7bcb6d57ee19b002342b39bbb39"
server_key_version: "encryptionkey@0"
I20260806 04:39:35.676256 2522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "86f24b7ccdd944e9ba5ca37395a45b15"
format_stamp: "Formatted at 2026-08-06 04:39:35 on dist-test-slave-vkjp"
server_key: "ebc70b9516332af2d9c394ad93297b89"
server_key_iv: "c5a5c7bcb6d57ee19b002342b39bbb39"
server_key_version: "encryptionkey@0"
I20260806 04:39:35.680035 2522 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:39:35.682653 2539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:35.684185 2522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:39:35.684326 2522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "86f24b7ccdd944e9ba5ca37395a45b15"
format_stamp: "Formatted at 2026-08-06 04:39:35 on dist-test-slave-vkjp"
server_key: "ebc70b9516332af2d9c394ad93297b89"
server_key_iv: "c5a5c7bcb6d57ee19b002342b39bbb39"
server_key_version: "encryptionkey@0"
I20260806 04:39:35.684468 2522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:35.696220 2522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:35.699862 2522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:35.700093 2522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:35.708068 2522 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:35253
I20260806 04:39:35.708096 2593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:35253 every 8 connection(s)
I20260806 04:39:35.709093 2522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:35.711907 2594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:35.716003 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 2522
I20260806 04:39:35.716097 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:35.716388 32410 external_mini_cluster.cc:1428] Setting key c1ed21bf3c1900d8f3e9be87b90351a3
I20260806 04:39:35.717868 2594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15: Bootstrap starting.
I20260806 04:39:35.720160 2594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:35.720939 2594 log.cc:824] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:35.722901 2594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15: No bootstrap required, opened a new log
Aug 06 04:39:35 dist-test-slave-vkjp krb5kdc[2507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:35.727334 2594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f24b7ccdd944e9ba5ca37395a45b15" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35253 } }
I20260806 04:39:35.727663 2594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:35.727710 2594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86f24b7ccdd944e9ba5ca37395a45b15, State: Initialized, Role: FOLLOWER
I20260806 04:39:35.728204 2594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f24b7ccdd944e9ba5ca37395a45b15" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35253 } }
I20260806 04:39:35.728358 2594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:35.728446 2594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:35.728564 2594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:35.729833 2594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f24b7ccdd944e9ba5ca37395a45b15" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35253 } }
I20260806 04:39:35.730217 2594 leader_election.cc:302] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86f24b7ccdd944e9ba5ca37395a45b15; no voters:
I20260806 04:39:35.730605 2594 leader_election.cc:288] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:35.730719 2599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:35.730952 2599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [term 1 LEADER]: Becoming Leader. State: Replica: 86f24b7ccdd944e9ba5ca37395a45b15, State: Running, Role: LEADER
I20260806 04:39:35.731302 2599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f24b7ccdd944e9ba5ca37395a45b15" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35253 } }
I20260806 04:39:35.731895 2594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:35.732414 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:35.717918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.31.166.190:35253)
0806 04:39:35.718370 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:35.718385 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:35.718422 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:39:35.719079 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:35.719174 (+ 95us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:35.719508 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:35.719840 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:35.720736 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:35.721789 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:35.722668 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:35.723285 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:35.723461 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:35.726438 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:35.726444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:35.726458 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:35.726489 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:35.728689 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:35.729198 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:35.729203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:35.729207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:35.729321 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:35.729624 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:35.729626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:35.729627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:35.730088 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:35.730228 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:35.730628 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 04:39:35.731362 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":136,"threads_started":1}
I20260806 04:39:35.732981 2600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86f24b7ccdd944e9ba5ca37395a45b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f24b7ccdd944e9ba5ca37395a45b15" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35253 } } }
I20260806 04:39:35.733108 2600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:35.733332 2600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86f24b7ccdd944e9ba5ca37395a45b15. Latest consensus state: current_term: 1 leader_uuid: "86f24b7ccdd944e9ba5ca37395a45b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86f24b7ccdd944e9ba5ca37395a45b15" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35253 } } }
I20260806 04:39:35.733421 2600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:35.734755 2609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:35.737880 2609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:35.743670 2609 catalog_manager.cc:1347] Generated new cluster ID: 7c5ef748029241e1bbc80771d520d138
I20260806 04:39:35.743747 2609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:35.760701 2609 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:35.762049 2609 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:35.772512 2609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86f24b7ccdd944e9ba5ca37395a45b15: Generated new TSK 0
I20260806 04:39:35.773362 2609 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:39:35.807386 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:35253
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:35.917970 2621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:35.918226 2621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:35.918269 2621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:35.922199 2621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:35.922255 2621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:35.922380 2621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:39:35.927498 2621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:35253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:35.928503 2621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:35.929523 2621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:39:35.939342 2621 server_base.cc:1034] running on GCE node
W20260806 04:39:35.939371 2628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:35.939368 2630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:35.939523 2627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:35.940038 2621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:35.940678 2621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:35.941847 2621 hybrid_clock.cc:648] HybridClock initialized: now 1785991175941811 us; error 49 us; skew 500 ppm
Aug 06 04:39:35 dist-test-slave-vkjp krb5kdc[2507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991175, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:35.944751 2621 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:39:35.945600 2621 webserver.cc:460] Webserver started at http://127.31.166.129:45123/ using document root <none> and password file <none>
I20260806 04:39:35.946130 2621 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:35.946190 2621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:35.946358 2621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:35.948486 2621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "a33bf62b344841d4ac8a5b65622aee13"
format_stamp: "Formatted at 2026-08-06 04:39:35 on dist-test-slave-vkjp"
server_key: "eae450dfe97ae34386d85aab8b2b66ef"
server_key_iv: "27ff200d70443229df7c700f19ca855f"
server_key_version: "encryptionkey@0"
I20260806 04:39:35.949064 2621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "a33bf62b344841d4ac8a5b65622aee13"
format_stamp: "Formatted at 2026-08-06 04:39:35 on dist-test-slave-vkjp"
server_key: "eae450dfe97ae34386d85aab8b2b66ef"
server_key_iv: "27ff200d70443229df7c700f19ca855f"
server_key_version: "encryptionkey@0"
I20260806 04:39:35.953464 2621 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260806 04:39:35.955897 2638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:35.957122 2621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:39:35.957356 2621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "a33bf62b344841d4ac8a5b65622aee13"
format_stamp: "Formatted at 2026-08-06 04:39:35 on dist-test-slave-vkjp"
server_key: "eae450dfe97ae34386d85aab8b2b66ef"
server_key_iv: "27ff200d70443229df7c700f19ca855f"
server_key_version: "encryptionkey@0"
I20260806 04:39:35.957479 2621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:35.975880 2621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:35.979162 2621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:35.979364 2621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:35.979964 2621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:35.980777 2621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:35.980849 2621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:35.980921 2621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:35.980969 2621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:35.990119 2621 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:44041
I20260806 04:39:35.990150 2759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:44041 every 8 connection(s)
I20260806 04:39:35.991021 2621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:39:35.993602 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 2621
I20260806 04:39:35.993736 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:39:35.994041 32410 external_mini_cluster.cc:1428] Setting key c0ce7af5c350c969acf27081a1014cc5
Aug 06 04:39:35 dist-test-slave-vkjp krb5kdc[2507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991175, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
I20260806 04:39:36.007325 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:35.992959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:33573 (local address 127.31.166.190:35253)
0806 04:39:35.993124 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:35.993127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:35.994038 (+ 911us) server_negotiation.cc:407] Connection header received
0806 04:39:35.995661 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:35.995664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:35.995721 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:35.995825 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:35.997112 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:35.997689 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:35.999603 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:35.999955 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:36.000063 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:36.003276 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:36.003279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:36.003282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:36.003309 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:36.004856 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:36.005384 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:36.005387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:36.005388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:36.005474 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:36.005799 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:36.005801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:36.005802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:36.006018 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:36.006126 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:36.006835 (+ 709us) server_negotiation.cc:299] Negotiation successful
0806 04:39:36.007020 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 04:39:36.008319 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:35.993357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:35253 (local address 127.31.166.129:33573)
0806 04:39:35.993846 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:35.993902 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:35.995433 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:35.996027 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:35.996034 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:35.996411 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:35.996903 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:35.996917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:35.997832 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:35.997838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:35.999444 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:35.999459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:36.000143 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:36.000149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:36.000393 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:36.001156 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:36.001184 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:36.003030 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:36.004984 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:36.004990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:36.005004 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:36.005277 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:36.005567 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:36.005569 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:36.005570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:36.005712 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:36.006193 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:36.006200 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:36.006532 (+ 332us) client_negotiation.cc:769] Sending connection context
0806 04:39:36.006705 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 04:39:36.007137 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":148,"threads_started":1}
I20260806 04:39:36.009690 2760 heartbeater.cc:347] Connected to a master server at 127.31.166.190:35253
I20260806 04:39:36.009950 2760 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:36.010588 2760 heartbeater.cc:511] Master 127.31.166.190:35253 requested a full tablet report, sending...
I20260806 04:39:36.012351 2556 ts_manager.cc:194] Registered new tserver with Master: a33bf62b344841d4ac8a5b65622aee13 (127.31.166.129:44041)
I20260806 04:39:36.013559 2556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:33573
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:39:36.027235 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:35253
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:36.147301 2766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:36.147576 2766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:36.147663 2766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:36.151688 2766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:36.151747 2766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:36.151851 2766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:39:36.157114 2766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:35253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:36.158207 2766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:36.159278 2766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:36.169020 2774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:36.169083 2772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:36.169106 2766 server_base.cc:1034] running on GCE node
W20260806 04:39:36.169008 2771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:36.169818 2766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:36.170439 2766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:36.171617 2766 hybrid_clock.cc:648] HybridClock initialized: now 1785991176171583 us; error 43 us; skew 500 ppm
Aug 06 04:39:36 dist-test-slave-vkjp krb5kdc[2507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991176, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:36.174610 2766 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:39:36.175668 2766 webserver.cc:460] Webserver started at http://127.31.166.130:38523/ using document root <none> and password file <none>
I20260806 04:39:36.176270 2766 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:36.176381 2766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:36.176633 2766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:36.178777 2766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "5f9935d1764f4494b65a2650c5297a49"
format_stamp: "Formatted at 2026-08-06 04:39:36 on dist-test-slave-vkjp"
server_key: "c69c9dd3bb54ecdafc35eb12cc27da03"
server_key_iv: "e7763992753c3bd8161cd6f17153f279"
server_key_version: "encryptionkey@0"
I20260806 04:39:36.179427 2766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "5f9935d1764f4494b65a2650c5297a49"
format_stamp: "Formatted at 2026-08-06 04:39:36 on dist-test-slave-vkjp"
server_key: "c69c9dd3bb54ecdafc35eb12cc27da03"
server_key_iv: "e7763992753c3bd8161cd6f17153f279"
server_key_version: "encryptionkey@0"
I20260806 04:39:36.183408 2766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:39:36.185945 2781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:36.187260 2766 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:36.187446 2766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "5f9935d1764f4494b65a2650c5297a49"
format_stamp: "Formatted at 2026-08-06 04:39:36 on dist-test-slave-vkjp"
server_key: "c69c9dd3bb54ecdafc35eb12cc27da03"
server_key_iv: "e7763992753c3bd8161cd6f17153f279"
server_key_version: "encryptionkey@0"
I20260806 04:39:36.187570 2766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:36.207378 2766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:36.210641 2766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:36.210989 2766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:36.211933 2766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:36.212916 2766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:36.212998 2766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:36.213075 2766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:36.213110 2766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:36.223035 2766 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:42043
I20260806 04:39:36.223061 2902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:42043 every 8 connection(s)
I20260806 04:39:36.224040 2766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:39:36.224267 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 2766
I20260806 04:39:36.224385 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:39:36.224709 32410 external_mini_cluster.cc:1428] Setting key ecb6b7f9917ec6f0d61fc138e60df029
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
Aug 06 04:39:36 dist-test-slave-vkjp krb5kdc[2507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991176, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
I20260806 04:39:36.240494 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:36.226041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:59031 (local address 127.31.166.190:35253)
0806 04:39:36.226191 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:36.226196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:36.227341 (+ 1145us) server_negotiation.cc:407] Connection header received
0806 04:39:36.228988 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:36.228994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:36.229065 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:36.229165 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:36.230670 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:36.231415 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:36.232866 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:36.233122 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:36.233246 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:36.236370 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:36.236374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:36.236376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:36.236405 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:36.238001 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:36.238533 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:36.238542 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:36.238543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:36.238670 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:36.239201 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:36.239206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:36.239209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:36.239468 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:36.239593 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:36.240147 (+ 554us) server_negotiation.cc:299] Negotiation successful
0806 04:39:36.240300 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:39:36.241469 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:36.226586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:35253 (local address 127.31.166.130:59031)
0806 04:39:36.227163 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:36.227223 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:36.228775 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:36.229328 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:36.229334 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:36.229797 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:36.230493 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:36.230520 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:36.231583 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:36.231587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:36.232731 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:36.232742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:36.233292 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:36.233296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:36.233496 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:36.234281 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:36.234301 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:36.236126 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:36.238146 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:36.238151 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:36.238162 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:36.238415 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:36.238823 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:36.238827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:36.238829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:36.239043 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:36.239593 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:36.239602 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:36.239877 (+ 275us) client_negotiation.cc:769] Sending connection context
0806 04:39:36.240088 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 04:39:36.240438 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":131,"threads_started":1}
I20260806 04:39:36.242812 2903 heartbeater.cc:347] Connected to a master server at 127.31.166.190:35253
I20260806 04:39:36.243098 2903 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:36.243728 2903 heartbeater.cc:511] Master 127.31.166.190:35253 requested a full tablet report, sending...
I20260806 04:39:36.244961 2556 ts_manager.cc:194] Registered new tserver with Master: 5f9935d1764f4494b65a2650c5297a49 (127.31.166.130:42043)
I20260806 04:39:36.245651 2556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:59031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:39:36.254386 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:35253
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:40489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:36.363263 2908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:36.363515 2908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:36.363559 2908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:36.367574 2908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:36.367630 2908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:36.367733 2908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:39:36.372854 2908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:40489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:35253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:36.373840 2908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:36.374821 2908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:36.384152 2914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:36.384163 2913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:36.384140 2916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:36.384397 2908 server_base.cc:1034] running on GCE node
I20260806 04:39:36.384902 2908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:36.385512 2908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:36.386685 2908 hybrid_clock.cc:648] HybridClock initialized: now 1785991176386672 us; error 23 us; skew 500 ppm
Aug 06 04:39:36 dist-test-slave-vkjp krb5kdc[2507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991176, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:36.389586 2908 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:39:36.390434 2908 webserver.cc:460] Webserver started at http://127.31.166.131:35849/ using document root <none> and password file <none>
I20260806 04:39:36.391036 2908 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:36.391162 2908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:36.391402 2908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:36.393501 2908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "66001c98a4064764bee30c924d4c6e36"
format_stamp: "Formatted at 2026-08-06 04:39:36 on dist-test-slave-vkjp"
server_key: "d0fbd493e645590c97c2e089a2e900cb"
server_key_iv: "315169ca8f0131cd04974473419e3b71"
server_key_version: "encryptionkey@0"
I20260806 04:39:36.394058 2908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "66001c98a4064764bee30c924d4c6e36"
format_stamp: "Formatted at 2026-08-06 04:39:36 on dist-test-slave-vkjp"
server_key: "d0fbd493e645590c97c2e089a2e900cb"
server_key_iv: "315169ca8f0131cd04974473419e3b71"
server_key_version: "encryptionkey@0"
I20260806 04:39:36.397657 2908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 04:39:36.400112 2923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:36.401321 2908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:36.401456 2908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "66001c98a4064764bee30c924d4c6e36"
format_stamp: "Formatted at 2026-08-06 04:39:36 on dist-test-slave-vkjp"
server_key: "d0fbd493e645590c97c2e089a2e900cb"
server_key_iv: "315169ca8f0131cd04974473419e3b71"
server_key_version: "encryptionkey@0"
I20260806 04:39:36.401569 2908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:36.414196 2908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:36.417321 2908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:36.417518 2908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:36.418071 2908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:36.418857 2908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:36.418936 2908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:36.419008 2908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:36.419061 2908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:36.427940 2908 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:45411
I20260806 04:39:36.427961 3041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:45411 every 8 connection(s)
I20260806 04:39:36.428805 2908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:39:36.430156 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 2908
I20260806 04:39:36.430284 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:39:36.430620 32410 external_mini_cluster.cc:1428] Setting key fad1feb9cc6f7326bde8caa388c32ae1
Aug 06 04:39:36 dist-test-slave-vkjp krb5kdc[2507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991176, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:36.443115 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:36.430736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:57081 (local address 127.31.166.190:35253)
0806 04:39:36.430896 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:36.430901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:36.432330 (+ 1429us) server_negotiation.cc:407] Connection header received
0806 04:39:36.433829 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:36.433832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:36.433887 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:36.433971 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:36.435175 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:36.435709 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:36.436829 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:36.437060 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:36.437170 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:36.439584 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:36.439587 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:36.439590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:36.439622 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:36.441063 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:36.441580 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:36.441583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:36.441585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:36.441657 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:36.442004 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:36.442007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:36.442009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:36.442187 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:36.442294 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:36.442778 (+ 484us) server_negotiation.cc:299] Negotiation successful
0806 04:39:36.442898 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 04:39:36.443955 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:36.431216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:35253 (local address 127.31.166.131:57081)
0806 04:39:36.432174 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:36.432233 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:36.433631 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:36.434151 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:36.434160 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:36.434543 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:36.434996 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:36.435008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:36.435836 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:36.435839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:36.436703 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:36.436712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:36.437207 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:36.437211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:36.437380 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:36.437984 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:36.438000 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:36.439425 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:36.441183 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:36.441188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:36.441201 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:36.441464 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:36.441749 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:36.441752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:36.441754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:36.441899 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:36.442291 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:36.442298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:36.442541 (+ 243us) client_negotiation.cc:769] Sending connection context
0806 04:39:36.442719 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 04:39:36.443025 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":182,"threads_started":1}
I20260806 04:39:36.445089 3042 heartbeater.cc:347] Connected to a master server at 127.31.166.190:35253
I20260806 04:39:36.445389 3042 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:36.445902 3042 heartbeater.cc:511] Master 127.31.166.190:35253 requested a full tablet report, sending...
I20260806 04:39:36.446923 2556 ts_manager.cc:194] Registered new tserver with Master: 66001c98a4064764bee30c924d4c6e36 (127.31.166.131:45411)
I20260806 04:39:36.447606 2556 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:57081
I20260806 04:39:36.456848 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 04:39:36.469756 2597 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:39:36.468236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.31.166.190:35253)
0806 04:39:36.468454 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:36.468457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:36.468477 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:39:36.468619 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:36.468622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:36.468666 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:36.468738 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:36.469606 (+ 868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56618: BlockingRecv error: recv got EOF from 127.0.0.1:56618 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 04:39:36.471866 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 2621
I20260806 04:39:36.479398 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 2766
I20260806 04:39:36.487650 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 2908
I20260806 04:39:36.494861 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 2522
2026-08-06T04:39:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2185 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:39:36 dist-test-slave-vkjp krb5kdc[3055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:39:36 dist-test-slave-vkjp krb5kdc[3055](info): set up 2 sockets
Aug 06 04:39:36 dist-test-slave-vkjp krb5kdc[3055](info): commencing operation
krb5kdc: starting...
W20260806 04:39:37.586274 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:39:37 dist-test-slave-vkjp krb5kdc[3055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:39:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:39:37.718502 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:46499
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:39285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:46499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:37.829828 3069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:37.830149 3069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:37.830233 3069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:37.834295 3069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:37.834403 3069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:37.834445 3069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:37.834497 3069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:37.834550 3069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:39:37.839952 3069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:39285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:46499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:46499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:37.841148 3069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:37.842211 3069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:37.850852 3074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:37.850868 3077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:37.850860 3075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:37.850958 3069 server_base.cc:1034] running on GCE node
I20260806 04:39:37.851699 3069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:37.852788 3069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:37.853953 3069 hybrid_clock.cc:648] HybridClock initialized: now 1785991177853938 us; error 28 us; skew 500 ppm
Aug 06 04:39:37 dist-test-slave-vkjp krb5kdc[3055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991177, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:37.857158 3069 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:39:37.858134 3069 webserver.cc:460] Webserver started at http://127.31.166.190:46427/ using document root <none> and password file <none>
I20260806 04:39:37.858989 3069 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:37.859127 3069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:37.859391 3069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:37.861594 3069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "2057ae892f8f466a9ae4f74061e48397"
format_stamp: "Formatted at 2026-08-06 04:39:37 on dist-test-slave-vkjp"
server_key: "5a21ba177a5d20c7a36c969ceec289ac"
server_key_iv: "8207853228a48171f26b819ee4e0adc4"
server_key_version: "encryptionkey@0"
I20260806 04:39:37.862124 3069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "2057ae892f8f466a9ae4f74061e48397"
format_stamp: "Formatted at 2026-08-06 04:39:37 on dist-test-slave-vkjp"
server_key: "5a21ba177a5d20c7a36c969ceec289ac"
server_key_iv: "8207853228a48171f26b819ee4e0adc4"
server_key_version: "encryptionkey@0"
I20260806 04:39:37.865811 3069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 04:39:37.868194 3084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:37.869467 3069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:39:37.869571 3069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "2057ae892f8f466a9ae4f74061e48397"
format_stamp: "Formatted at 2026-08-06 04:39:37 on dist-test-slave-vkjp"
server_key: "5a21ba177a5d20c7a36c969ceec289ac"
server_key_iv: "8207853228a48171f26b819ee4e0adc4"
server_key_version: "encryptionkey@0"
I20260806 04:39:37.869660 3069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:37.885630 3069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:37.888720 3069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:37.888882 3069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:37.895896 3069 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:46499
I20260806 04:39:37.896253 3137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:46499 every 8 connection(s)
I20260806 04:39:37.896893 3069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:37.899761 3138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:37.905156 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 3069
I20260806 04:39:37.905251 3138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397: Bootstrap starting.
I20260806 04:39:37.905283 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:37.905580 32410 external_mini_cluster.cc:1428] Setting key 700b903d50770aed8946bcb6c4e8a386
I20260806 04:39:37.907989 3138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:37.908773 3138 log.cc:824] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:37.910789 3138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397: No bootstrap required, opened a new log
I20260806 04:39:37.914265 3138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057ae892f8f466a9ae4f74061e48397" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 46499 } }
I20260806 04:39:37.914595 3138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:37.914647 3138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2057ae892f8f466a9ae4f74061e48397, State: Initialized, Role: FOLLOWER
Aug 06 04:39:37 dist-test-slave-vkjp krb5kdc[3055](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:37.915198 3138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057ae892f8f466a9ae4f74061e48397" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 46499 } }
I20260806 04:39:37.915381 3138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:37.915459 3138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:37.915565 3138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:37.916778 3138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057ae892f8f466a9ae4f74061e48397" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 46499 } }
I20260806 04:39:37.917172 3138 leader_election.cc:302] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2057ae892f8f466a9ae4f74061e48397; no voters:
I20260806 04:39:37.917496 3138 leader_election.cc:288] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:37.917625 3143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:37.917850 3143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [term 1 LEADER]: Becoming Leader. State: Replica: 2057ae892f8f466a9ae4f74061e48397, State: Running, Role: LEADER
I20260806 04:39:37.918207 3143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057ae892f8f466a9ae4f74061e48397" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 46499 } }
I20260806 04:39:37.918952 3138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:37.919615 3144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2057ae892f8f466a9ae4f74061e48397. Latest consensus state: current_term: 1 leader_uuid: "2057ae892f8f466a9ae4f74061e48397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057ae892f8f466a9ae4f74061e48397" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 46499 } } }
I20260806 04:39:37.919768 3144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:37.920019 3145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2057ae892f8f466a9ae4f74061e48397" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2057ae892f8f466a9ae4f74061e48397" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 46499 } } }
I20260806 04:39:37.920132 3145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:37.920183 3149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:37.921972 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:37.906900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.31.166.190:46499)
0806 04:39:37.907391 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:37.907412 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:37.907450 (+ 38us) server_negotiation.cc:407] Connection header received
0806 04:39:37.908112 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:37.908136 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:37.908484 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:37.908758 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:37.909725 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:37.910896 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:37.911977 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:37.912505 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:37.912697 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:37.915680 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:37.915687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:37.915705 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:37.915743 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:37.917833 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:37.918406 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:37.918411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:37.918418 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:37.918561 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:37.918910 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:37.918913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:37.918915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:37.919499 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:37.920497 (+ 998us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:37.920646 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 04:39:37.920963 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":192,"threads_started":1}
I20260806 04:39:37.924180 3149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:37.925177 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:39:37.929926 3149 catalog_manager.cc:1347] Generated new cluster ID: bdd491be422d43bda39ac6c57d99d591
I20260806 04:39:37.930009 3149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:37.947759 3149 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:37.949018 3149 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:37.950265 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:37.944096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.31.166.190:46499)
0806 04:39:37.944266 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:37.944271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:37.944561 (+ 290us) server_negotiation.cc:407] Connection header received
0806 04:39:37.944766 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:37.944769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:37.944831 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:37.944925 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:37.945534 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:37.946100 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:37.947442 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:37.947794 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:37.947914 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:37.949072 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:37.949077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:37.949082 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:37.949708 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:37.949749 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:37.949772 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:37.949884 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:37.950026 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 04:39:37.950116 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 04:39:37.957150 3149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2057ae892f8f466a9ae4f74061e48397: Generated new TSK 0
I20260806 04:39:37.957904 3149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:39:37.965723 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 3069
2026-08-06T04:39:37Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1469 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:39:38 dist-test-slave-vkjp krb5kdc[3173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:39:38 dist-test-slave-vkjp krb5kdc[3173](info): set up 2 sockets
Aug 06 04:39:38 dist-test-slave-vkjp krb5kdc[3173](info): commencing operation
krb5kdc: starting...
W20260806 04:39:39.060142 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:39:39 dist-test-slave-vkjp krb5kdc[3173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:39:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:39:39.185670 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:43669
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:43669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:39.295495 3188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:39.295797 3188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:39.295871 3188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:39.299887 3188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:39.299974 3188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:39.300012 3188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:39.300067 3188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:39.300102 3188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:39:39.305563 3188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:43669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:43669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:39.306716 3188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:39.307801 3188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:39.316445 3194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:39.316461 3193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:39.316473 3196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:39.316697 3188 server_base.cc:1034] running on GCE node
I20260806 04:39:39.317373 3188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:39.318476 3188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:39.319682 3188 hybrid_clock.cc:648] HybridClock initialized: now 1785991179319639 us; error 58 us; skew 500 ppm
Aug 06 04:39:39 dist-test-slave-vkjp krb5kdc[3173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991179, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:39.322530 3188 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:39:39.323503 3188 webserver.cc:460] Webserver started at http://127.31.166.190:42411/ using document root <none> and password file <none>
I20260806 04:39:39.324050 3188 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:39.324110 3188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:39.324275 3188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:39.326236 3188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "55e90175039045439c5f11a1bd3b75b5"
format_stamp: "Formatted at 2026-08-06 04:39:39 on dist-test-slave-vkjp"
server_key: "3b006e17124e8e940b16acbad274c4a9"
server_key_iv: "08f288f99a3741c98cb2cf1b314eb75e"
server_key_version: "encryptionkey@0"
I20260806 04:39:39.326766 3188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "55e90175039045439c5f11a1bd3b75b5"
format_stamp: "Formatted at 2026-08-06 04:39:39 on dist-test-slave-vkjp"
server_key: "3b006e17124e8e940b16acbad274c4a9"
server_key_iv: "08f288f99a3741c98cb2cf1b314eb75e"
server_key_version: "encryptionkey@0"
I20260806 04:39:39.330463 3188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 04:39:39.333020 3204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:39.334461 3188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:39:39.334616 3188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "55e90175039045439c5f11a1bd3b75b5"
format_stamp: "Formatted at 2026-08-06 04:39:39 on dist-test-slave-vkjp"
server_key: "3b006e17124e8e940b16acbad274c4a9"
server_key_iv: "08f288f99a3741c98cb2cf1b314eb75e"
server_key_version: "encryptionkey@0"
I20260806 04:39:39.334744 3188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:39.348817 3188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:39.352171 3188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:39.352386 3188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:39.360447 3188 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:43669
I20260806 04:39:39.360754 3257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:43669 every 8 connection(s)
I20260806 04:39:39.361477 3188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:39.362123 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 3188
I20260806 04:39:39.362263 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:39.362550 32410 external_mini_cluster.cc:1428] Setting key 112a443d3864a4be213c8690f85eee83
I20260806 04:39:39.364768 3258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:39.370162 3258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5: Bootstrap starting.
I20260806 04:39:39.372357 3258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:39.373122 3258 log.cc:824] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:39:39 dist-test-slave-vkjp krb5kdc[3173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991179, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:39.375504 3258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5: No bootstrap required, opened a new log
I20260806 04:39:39.379443 3258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e90175039045439c5f11a1bd3b75b5" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43669 } }
I20260806 04:39:39.379650 3258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:39.379719 3258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e90175039045439c5f11a1bd3b75b5, State: Initialized, Role: FOLLOWER
I20260806 04:39:39.380198 3258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e90175039045439c5f11a1bd3b75b5" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43669 } }
I20260806 04:39:39.380321 3258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:39.380373 3261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:39.363982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50136 (local address 127.31.166.190:43669)
0806 04:39:39.364430 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:39.364445 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:39.364481 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:39:39.365117 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:39.365135 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:39.365475 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:39.365825 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:39.366692 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:39.367725 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:39.368601 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:39.369063 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:39.369228 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:39.375541 (+ 6313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:39.375546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:39.375576 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:39.375612 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:39.377326 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:39.377845 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:39.377849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:39.377853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:39.377961 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:39.378239 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:39.378241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:39.378242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:39.378657 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:39.378810 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:39.379182 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 04:39:39.379524 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":137,"threads_started":1}
I20260806 04:39:39.380407 3258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:39.380555 3258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:39.381488 3258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e90175039045439c5f11a1bd3b75b5" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43669 } }
I20260806 04:39:39.381835 3258 leader_election.cc:302] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55e90175039045439c5f11a1bd3b75b5; no voters:
I20260806 04:39:39.382174 3258 leader_election.cc:288] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:39.382361 3263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:39.382893 3263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [term 1 LEADER]: Becoming Leader. State: Replica: 55e90175039045439c5f11a1bd3b75b5, State: Running, Role: LEADER
I20260806 04:39:39.383213 3263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e90175039045439c5f11a1bd3b75b5" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43669 } }
I20260806 04:39:39.383455 3258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:39.384754 3265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55e90175039045439c5f11a1bd3b75b5. Latest consensus state: current_term: 1 leader_uuid: "55e90175039045439c5f11a1bd3b75b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e90175039045439c5f11a1bd3b75b5" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43669 } } }
I20260806 04:39:39.384900 3265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:39.385250 3264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55e90175039045439c5f11a1bd3b75b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e90175039045439c5f11a1bd3b75b5" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43669 } } }
I20260806 04:39:39.385370 3264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:39.385308 3276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:39.388331 3276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:39.393591 3276 catalog_manager.cc:1347] Generated new cluster ID: 4dd72c83807141d888483534c2699711
I20260806 04:39:39.393657 3276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:39.410543 3276 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:39.411842 3276 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:39.428987 3276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55e90175039045439c5f11a1bd3b75b5: Generated new TSK 0
I20260806 04:39:39.429731 3276 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:39:39.464246 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43669
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:39.582008 3285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:39.582273 3285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:39.582315 3285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:39.587001 3285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:39.587118 3285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:39.587227 3285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:39:39.594681 3285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:39.595877 3285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:39.596940 3285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:39.606678 3292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:39.606838 3285 server_base.cc:1034] running on GCE node
W20260806 04:39:39.606680 3291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:39.607025 3294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:39.607468 3285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:39.608093 3285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:39.609280 3285 hybrid_clock.cc:648] HybridClock initialized: now 1785991179609267 us; error 24 us; skew 500 ppm
Aug 06 04:39:39 dist-test-slave-vkjp krb5kdc[3173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991179, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:39.612106 3285 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:39:39.612934 3285 webserver.cc:460] Webserver started at http://127.31.166.129:43945/ using document root <none> and password file <none>
I20260806 04:39:39.613482 3285 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:39.613571 3285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:39.613806 3285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:39.615897 3285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "5b6e2de2a3104d5294771daf70500c01"
format_stamp: "Formatted at 2026-08-06 04:39:39 on dist-test-slave-vkjp"
server_key: "16e8646324297a48a6da132fe4e7a106"
server_key_iv: "5cdc0c76af907754dc2cf0ae7522ba09"
server_key_version: "encryptionkey@0"
I20260806 04:39:39.616470 3285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "5b6e2de2a3104d5294771daf70500c01"
format_stamp: "Formatted at 2026-08-06 04:39:39 on dist-test-slave-vkjp"
server_key: "16e8646324297a48a6da132fe4e7a106"
server_key_iv: "5cdc0c76af907754dc2cf0ae7522ba09"
server_key_version: "encryptionkey@0"
I20260806 04:39:39.620214 3285 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:39:39.622839 3302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:39.624121 3285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:39.624254 3285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "5b6e2de2a3104d5294771daf70500c01"
format_stamp: "Formatted at 2026-08-06 04:39:39 on dist-test-slave-vkjp"
server_key: "16e8646324297a48a6da132fe4e7a106"
server_key_iv: "5cdc0c76af907754dc2cf0ae7522ba09"
server_key_version: "encryptionkey@0"
I20260806 04:39:39.624377 3285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:39.640021 3285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:39.643607 3285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:39.643832 3285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:39.644462 3285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:39.645318 3285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:39.645363 3285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:39.645430 3285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:39.645463 3285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:39.654858 3285 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:42643
I20260806 04:39:39.654876 3421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:42643 every 8 connection(s)
I20260806 04:39:39.655840 3285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:39:39.661087 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 3285
I20260806 04:39:39.661234 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:39:39.661550 32410 external_mini_cluster.cc:1428] Setting key 3cc24e490e0350628cf03905cecd8b2c
Aug 06 04:39:39 dist-test-slave-vkjp krb5kdc[3173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991179, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:39.671139 3261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:39.657751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:42941 (local address 127.31.166.190:43669)
0806 04:39:39.657896 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:39.657900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:39.658685 (+ 785us) server_negotiation.cc:407] Connection header received
0806 04:39:39.660190 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:39.660193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:39.660253 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:39.660363 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:39.661889 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:39.662561 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:39.664162 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:39.664383 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:39.664474 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:39.667213 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:39.667219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:39.667226 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:39.667260 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:39.668969 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:39.669482 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:39.669486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:39.669488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:39.669586 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:39.669921 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:39.669925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:39.669927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:39.670173 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:39.670284 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:39.670750 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 04:39:39.670889 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
I20260806 04:39:39.671888 3424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:39.658079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43669 (local address 127.31.166.129:42941)
0806 04:39:39.658536 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:39.658578 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:39.659979 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:39.660547 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:39.660553 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:39.661030 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:39.661730 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:39.661751 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:39.662694 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:39.662697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:39.664039 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:39.664054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:39.664546 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:39.664553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:39.664782 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:39.665462 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:39.665479 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:39.666982 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:39.669085 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:39.669090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:39.669100 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:39.669344 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:39.669683 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:39.669685 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:39.669686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:39.669812 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:39.670279 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:39.670284 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:39.670509 (+ 225us) client_negotiation.cc:769] Sending connection context
0806 04:39:39.670650 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 04:39:39.670882 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":121,"threads_started":1}
I20260806 04:39:39.673188 3422 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43669
I20260806 04:39:39.673417 3422 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:39.673887 3422 heartbeater.cc:511] Master 127.31.166.190:43669 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
I20260806 04:39:39.675539 3221 ts_manager.cc:194] Registered new tserver with Master: 5b6e2de2a3104d5294771daf70500c01 (127.31.166.129:42643)
I20260806 04:39:39.676834 3221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:42941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:39:39.690658 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43669
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:39.802310 3428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:39.802596 3428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:39.802641 3428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:39.806802 3428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:39.806870 3428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:39.806970 3428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:39:39.812361 3428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:39.813377 3428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:39.814402 3428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:39.824249 3433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:39.824404 3428 server_base.cc:1034] running on GCE node
W20260806 04:39:39.824254 3434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:39.824254 3436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:39.825030 3428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:39.825788 3428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:39.827170 3428 hybrid_clock.cc:648] HybridClock initialized: now 1785991179827123 us; error 67 us; skew 500 ppm
Aug 06 04:39:39 dist-test-slave-vkjp krb5kdc[3173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991179, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:39.830001 3428 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:39:39.830930 3428 webserver.cc:460] Webserver started at http://127.31.166.130:43867/ using document root <none> and password file <none>
I20260806 04:39:39.831549 3428 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:39.831642 3428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:39.831876 3428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:39.833916 3428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "ae3ebd6c2bb849f88fea40edfe78049e"
format_stamp: "Formatted at 2026-08-06 04:39:39 on dist-test-slave-vkjp"
server_key: "c66fa6d2a00686ede134adeae65e2d55"
server_key_iv: "ac472f298ba3e7f98e3dd395a87700ba"
server_key_version: "encryptionkey@0"
I20260806 04:39:39.834489 3428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "ae3ebd6c2bb849f88fea40edfe78049e"
format_stamp: "Formatted at 2026-08-06 04:39:39 on dist-test-slave-vkjp"
server_key: "c66fa6d2a00686ede134adeae65e2d55"
server_key_iv: "ac472f298ba3e7f98e3dd395a87700ba"
server_key_version: "encryptionkey@0"
I20260806 04:39:39.838295 3428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 04:39:39.841027 3444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:39.842274 3428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:39:39.842445 3428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "ae3ebd6c2bb849f88fea40edfe78049e"
format_stamp: "Formatted at 2026-08-06 04:39:39 on dist-test-slave-vkjp"
server_key: "c66fa6d2a00686ede134adeae65e2d55"
server_key_iv: "ac472f298ba3e7f98e3dd395a87700ba"
server_key_version: "encryptionkey@0"
I20260806 04:39:39.842569 3428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:39.857604 3428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:39.860971 3428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:39.861205 3428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:39.861817 3428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:39.862650 3428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:39.862721 3428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:39.862794 3428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:39.862847 3428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:39.871794 3428 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:43783
I20260806 04:39:39.871824 3567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:43783 every 8 connection(s)
I20260806 04:39:39.872741 3428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:39:39.877170 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 3428
I20260806 04:39:39.877308 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:39:39.877663 32410 external_mini_cluster.cc:1428] Setting key ec458cf88a2cacc7cb1e87c0cc74077f
Aug 06 04:39:39 dist-test-slave-vkjp krb5kdc[3173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991179, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:39:39.889156 3261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:39.874653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:34375 (local address 127.31.166.190:43669)
0806 04:39:39.874816 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:39.874820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:39.875793 (+ 973us) server_negotiation.cc:407] Connection header received
0806 04:39:39.877276 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:39.877280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:39.877348 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:39.877480 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:39.878906 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:39.879657 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:39.881344 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:39.881572 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:39.881685 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:39.884799 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:39.884801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:39.884803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:39.884827 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:39.886536 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:39.887227 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:39.887230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:39.887231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:39.887329 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:39.887866 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:39.887868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:39.887870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:39.888072 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:39.888152 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:39.888912 (+ 760us) server_negotiation.cc:299] Negotiation successful
0806 04:39:39.889034 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 04:39:39.890182 3570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:39.875053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43669 (local address 127.31.166.130:34375)
0806 04:39:39.875626 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:39.875670 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:39.877007 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:39.877716 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:39.877723 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:39.878186 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:39.878725 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:39.878739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:39.879817 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:39.879823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:39.881217 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:39.881227 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:39.881725 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:39.881729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:39.881899 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:39.882541 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:39.882558 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:39.884613 (+ 2055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:39.886663 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:39.886670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:39.886689 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:39.887121 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:39.887479 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:39.887482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:39.887483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:39.887728 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:39.888196 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:39.888205 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:39.888570 (+ 365us) client_negotiation.cc:769] Sending connection context
0806 04:39:39.888797 (+ 227us) client_negotiation.cc:240] Negotiation successful
0806 04:39:39.889199 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":131,"threads_started":1}
I20260806 04:39:39.891459 3568 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
I20260806 04:39:39.891734 3568 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:39.892335 3568 heartbeater.cc:511] Master 127.31.166.190:43669 requested a full tablet report, sending...
I20260806 04:39:39.893451 3221 ts_manager.cc:194] Registered new tserver with Master: ae3ebd6c2bb849f88fea40edfe78049e (127.31.166.130:43783)
I20260806 04:39:39.894093 3221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:34375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:39:39.908417 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43669
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:43745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:40.021144 3573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:40.021399 3573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:40.021467 3573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:40.025547 3573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:40.025602 3573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:40.025693 3573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:39:40.031036 3573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:43745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:40.032071 3573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:40.033075 3573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:40.042603 3578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:40.042598 3579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:40.042840 3581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:40.043099 3573 server_base.cc:1034] running on GCE node
I20260806 04:39:40.043598 3573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:40.044406 3573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:40.045636 3573 hybrid_clock.cc:648] HybridClock initialized: now 1785991180045613 us; error 39 us; skew 500 ppm
Aug 06 04:39:40 dist-test-slave-vkjp krb5kdc[3173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991180, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:40.048655 3573 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:39:40.049579 3573 webserver.cc:460] Webserver started at http://127.31.166.131:37393/ using document root <none> and password file <none>
I20260806 04:39:40.050158 3573 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:40.050227 3573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:40.050419 3573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:40.052678 3573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "a34de851755d4b5db12b6b961400a0aa"
format_stamp: "Formatted at 2026-08-06 04:39:40 on dist-test-slave-vkjp"
server_key: "48b81d3c17184863ddc1063028b2bb41"
server_key_iv: "baec30d16103f90803318ea988f81044"
server_key_version: "encryptionkey@0"
I20260806 04:39:40.053269 3573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "a34de851755d4b5db12b6b961400a0aa"
format_stamp: "Formatted at 2026-08-06 04:39:40 on dist-test-slave-vkjp"
server_key: "48b81d3c17184863ddc1063028b2bb41"
server_key_iv: "baec30d16103f90803318ea988f81044"
server_key_version: "encryptionkey@0"
I20260806 04:39:40.057898 3573 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:39:40.060702 3588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:40.062038 3573 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:39:40.062189 3573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "a34de851755d4b5db12b6b961400a0aa"
format_stamp: "Formatted at 2026-08-06 04:39:40 on dist-test-slave-vkjp"
server_key: "48b81d3c17184863ddc1063028b2bb41"
server_key_iv: "baec30d16103f90803318ea988f81044"
server_key_version: "encryptionkey@0"
I20260806 04:39:40.062322 3573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:40.085808 3573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:40.089203 3573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:40.089426 3573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:40.090073 3573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:40.090917 3573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:40.090997 3573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:40.091068 3573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:40.091154 3573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:40.099833 3573 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:40481
I20260806 04:39:40.099869 3709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:40481 every 8 connection(s)
I20260806 04:39:40.100724 3573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:39:40.104970 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 3573
I20260806 04:39:40.105106 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:39:40.105460 32410 external_mini_cluster.cc:1428] Setting key 629237163d326249f7eb2c1a0298916b
Aug 06 04:39:40 dist-test-slave-vkjp krb5kdc[3173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991180, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:40.115265 3261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:40.102612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:46767 (local address 127.31.166.190:43669)
0806 04:39:40.102754 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:40.102758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:40.103578 (+ 820us) server_negotiation.cc:407] Connection header received
0806 04:39:40.105155 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:40.105160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:40.105214 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:40.105303 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:40.106625 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:40.107414 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:40.108775 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:40.109008 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:40.109097 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:40.111766 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:40.111770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:40.111771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:40.111797 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:40.113339 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:40.113912 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:40.113915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:40.113916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:40.113988 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:40.114274 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:40.114275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:40.114276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:40.114467 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:40.114544 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:40.114986 (+ 442us) server_negotiation.cc:299] Negotiation successful
0806 04:39:40.115139 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:39:40.116103 3712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:40.102960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43669 (local address 127.31.166.131:46767)
0806 04:39:40.103424 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:40.103466 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:40.104906 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:40.105518 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:40.105528 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:40.105954 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:40.106457 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:40.106475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:40.107563 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:40.107569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:40.108654 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:40.108665 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:40.109139 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:40.109142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:40.109319 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:40.109976 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:40.109991 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:40.111583 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:40.113435 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:40.113439 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:40.113450 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:40.113796 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:40.114067 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:40.114069 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:40.114070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:40.114199 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:40.114556 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:40.114560 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:40.114770 (+ 210us) client_negotiation.cc:769] Sending connection context
0806 04:39:40.114943 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 04:39:40.115262 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":137,"threads_started":1}
I20260806 04:39:40.117172 3710 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43669
I20260806 04:39:40.117393 3710 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:40.117806 3710 heartbeater.cc:511] Master 127.31.166.190:43669 requested a full tablet report, sending...
I20260806 04:39:40.118891 3221 ts_manager.cc:194] Registered new tserver with Master: a34de851755d4b5db12b6b961400a0aa (127.31.166.131:40481)
I20260806 04:39:40.119580 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:39:40.119591 3221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:46767
Aug 06 04:39:40 dist-test-slave-vkjp krb5kdc[3173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991180, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 04:39:40 dist-test-slave-vkjp krb5kdc[3173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991180, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:40.145434 3261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:40.137033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.31.166.190:43669)
0806 04:39:40.137271 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:40.137275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:40.137291 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:39:40.137478 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:40.137480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:40.137530 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:40.137615 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:40.138355 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:40.138931 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:40.139850 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:40.140070 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:40.140153 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:40.142456 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:40.142459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:40.142461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:40.142485 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:40.143970 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:40.144373 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:40.144376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:40.144377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:40.144448 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:40.144675 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:40.144677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:40.144678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:40.144885 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:40.144986 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:40.145182 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 04:39:40.145275 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
W20260806 04:39:40.145859 3221 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50138
I20260806 04:39:40.146821 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 3285
I20260806 04:39:40.153498 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 3428
I20260806 04:39:40.159719 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 3573
I20260806 04:39:40.167048 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 3188
2026-08-06T04:39:40Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2203 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:39:40 dist-test-slave-vkjp krb5kdc[3723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:39:40 dist-test-slave-vkjp krb5kdc[3723](info): set up 2 sockets
Aug 06 04:39:40 dist-test-slave-vkjp krb5kdc[3723](info): commencing operation
krb5kdc: starting...
W20260806 04:39:41.253978 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:39:41 dist-test-slave-vkjp krb5kdc[3723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:39:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:39:41.378480 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:37915
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:37915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:41.489877 3737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:41.490191 3737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:41.490274 3737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:41.494417 3737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:41.494496 3737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:41.494534 3737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:41.494590 3737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:41.494632 3737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:39:41.499897 3737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:37915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:37915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:41.500957 3737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:41.501986 3737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:41.510829 3745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:41.510766 3742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:41.510836 3743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:41.511270 3737 server_base.cc:1034] running on GCE node
I20260806 04:39:41.511981 3737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:41.513057 3737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:41.514263 3737 hybrid_clock.cc:648] HybridClock initialized: now 1785991181514251 us; error 31 us; skew 500 ppm
Aug 06 04:39:41 dist-test-slave-vkjp krb5kdc[3723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991181, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:41.517246 3737 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:39:41.518148 3737 webserver.cc:460] Webserver started at http://127.31.166.190:39615/ using document root <none> and password file <none>
I20260806 04:39:41.518687 3737 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:41.518749 3737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:41.518921 3737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:41.521062 3737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "c3c5bd7ed1c644a3a6c1dfdc542e6923"
format_stamp: "Formatted at 2026-08-06 04:39:41 on dist-test-slave-vkjp"
server_key: "74ddb30088edaa987d13a8e3e95eee20"
server_key_iv: "45030e21dc501b76e00b492033430419"
server_key_version: "encryptionkey@0"
I20260806 04:39:41.521605 3737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "c3c5bd7ed1c644a3a6c1dfdc542e6923"
format_stamp: "Formatted at 2026-08-06 04:39:41 on dist-test-slave-vkjp"
server_key: "74ddb30088edaa987d13a8e3e95eee20"
server_key_iv: "45030e21dc501b76e00b492033430419"
server_key_version: "encryptionkey@0"
I20260806 04:39:41.525449 3737 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:39:41.527997 3752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:41.529304 3737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:41.529536 3737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "c3c5bd7ed1c644a3a6c1dfdc542e6923"
format_stamp: "Formatted at 2026-08-06 04:39:41 on dist-test-slave-vkjp"
server_key: "74ddb30088edaa987d13a8e3e95eee20"
server_key_iv: "45030e21dc501b76e00b492033430419"
server_key_version: "encryptionkey@0"
I20260806 04:39:41.529649 3737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:41.555265 3737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:41.558710 3737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:41.558939 3737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:41.566280 3737 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:37915
I20260806 04:39:41.566309 3809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:37915 every 8 connection(s)
I20260806 04:39:41.567351 3737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:41.570008 3810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:41.575239 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 3737
I20260806 04:39:41.575305 3810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923: Bootstrap starting.
I20260806 04:39:41.575412 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:41.575672 32410 external_mini_cluster.cc:1428] Setting key 5ef7992aa2c780b2573982c9c374c40a
I20260806 04:39:41.577934 3810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:41.578799 3810 log.cc:824] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:41.581040 3810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923: No bootstrap required, opened a new log
I20260806 04:39:41.584923 3810 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c5bd7ed1c644a3a6c1dfdc542e6923" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37915 } }
I20260806 04:39:41.585143 3810 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:41.585211 3810 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3c5bd7ed1c644a3a6c1dfdc542e6923, State: Initialized, Role: FOLLOWER
Aug 06 04:39:41 dist-test-slave-vkjp krb5kdc[3723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:41.585669 3810 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c5bd7ed1c644a3a6c1dfdc542e6923" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37915 } }
I20260806 04:39:41.585789 3810 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:41.585870 3810 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:41.585981 3810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:41.587244 3810 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c5bd7ed1c644a3a6c1dfdc542e6923" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37915 } }
I20260806 04:39:41.587689 3810 leader_election.cc:302] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3c5bd7ed1c644a3a6c1dfdc542e6923; no voters:
I20260806 04:39:41.588058 3810 leader_election.cc:288] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:41.589550 3815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:41.589769 3815 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [term 1 LEADER]: Becoming Leader. State: Replica: c3c5bd7ed1c644a3a6c1dfdc542e6923, State: Running, Role: LEADER
I20260806 04:39:41.590108 3815 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c5bd7ed1c644a3a6c1dfdc542e6923" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37915 } }
I20260806 04:39:41.591353 3815 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3c5bd7ed1c644a3a6c1dfdc542e6923. Latest consensus state: current_term: 1 leader_uuid: "c3c5bd7ed1c644a3a6c1dfdc542e6923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c5bd7ed1c644a3a6c1dfdc542e6923" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37915 } } }
I20260806 04:39:41.591385 3810 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:41.591488 3815 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:41.591681 3816 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3c5bd7ed1c644a3a6c1dfdc542e6923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c5bd7ed1c644a3a6c1dfdc542e6923" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37915 } } }
I20260806 04:39:41.591774 3816 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:41.594128 3820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:41.594411 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:41.577165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.31.166.190:37915)
0806 04:39:41.577624 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:41.577643 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:41.577684 (+ 41us) server_negotiation.cc:407] Connection header received
0806 04:39:41.578531 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:41.578553 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:41.578951 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:41.579316 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:41.580174 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:41.581590 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:41.582500 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:41.583200 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:41.583361 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:41.586004 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:41.586011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:41.586027 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:41.586064 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:41.589357 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:41.589966 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:41.589971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:41.589977 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:41.590114 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:41.590930 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:41.590934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:41.590935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:41.591500 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:41.592089 (+ 589us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:41.592962 (+ 873us) server_negotiation.cc:299] Negotiation successful
0806 04:39:41.593261 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":146,"threads_started":1}
I20260806 04:39:41.597365 3820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:41.603755 3820 catalog_manager.cc:1347] Generated new cluster ID: 8df50fec47f049d381ca2be59d04afdf
I20260806 04:39:41.603844 3820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:41.615855 3820 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:41.617033 3820 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:41.622529 3820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3c5bd7ed1c644a3a6c1dfdc542e6923: Generated new TSK 0
I20260806 04:39:41.623205 3820 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:39:41.654564 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:37915
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:41.779057 3837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:41.779366 3837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:41.779450 3837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:41.783638 3837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:41.783715 3837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:41.783803 3837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:39:41.788919 3837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:37915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:41.789958 3837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:41.791159 3837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:41.800892 3844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:41.800894 3843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:41.800879 3846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:41.801080 3837 server_base.cc:1034] running on GCE node
I20260806 04:39:41.801643 3837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:41.802297 3837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:41.803480 3837 hybrid_clock.cc:648] HybridClock initialized: now 1785991181803464 us; error 30 us; skew 500 ppm
Aug 06 04:39:41 dist-test-slave-vkjp krb5kdc[3723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991181, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:41.806249 3837 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:39:41.807150 3837 webserver.cc:460] Webserver started at http://127.31.166.129:34291/ using document root <none> and password file <none>
I20260806 04:39:41.807659 3837 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:41.807716 3837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:41.807874 3837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:41.809805 3837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "b6af462da508473aa80b99f8b3434b8a"
format_stamp: "Formatted at 2026-08-06 04:39:41 on dist-test-slave-vkjp"
server_key: "2967babafd2df6e3b0a38ab385ae71db"
server_key_iv: "5aee8de6b796829a12e79b53adcdd836"
server_key_version: "encryptionkey@0"
I20260806 04:39:41.810694 3837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "b6af462da508473aa80b99f8b3434b8a"
format_stamp: "Formatted at 2026-08-06 04:39:41 on dist-test-slave-vkjp"
server_key: "2967babafd2df6e3b0a38ab385ae71db"
server_key_iv: "5aee8de6b796829a12e79b53adcdd836"
server_key_version: "encryptionkey@0"
I20260806 04:39:41.814707 3837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260806 04:39:41.817211 3853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:41.818557 3837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:39:41.818711 3837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "b6af462da508473aa80b99f8b3434b8a"
format_stamp: "Formatted at 2026-08-06 04:39:41 on dist-test-slave-vkjp"
server_key: "2967babafd2df6e3b0a38ab385ae71db"
server_key_iv: "5aee8de6b796829a12e79b53adcdd836"
server_key_version: "encryptionkey@0"
I20260806 04:39:41.818836 3837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:41.836254 3837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:41.839787 3837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:41.840008 3837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:41.840631 3837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:41.841506 3837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:41.841557 3837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:41.841631 3837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:41.841666 3837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:41.851754 3837 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:42525
I20260806 04:39:41.851783 3972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:42525 every 8 connection(s)
I20260806 04:39:41.852705 3837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:39:41.861831 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 3837
I20260806 04:39:41.861972 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
Aug 06 04:39:41 dist-test-slave-vkjp krb5kdc[3723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991181, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:41.862347 32410 external_mini_cluster.cc:1428] Setting key 034d9090d707dcc99a89a099af845bf1
I20260806 04:39:41.866869 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:41.854507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:53643 (local address 127.31.166.190:37915)
0806 04:39:41.854666 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:41.854671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:41.855682 (+ 1011us) server_negotiation.cc:407] Connection header received
0806 04:39:41.857147 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:41.857150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:41.857195 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:41.857277 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:41.858403 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:41.858916 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:41.860076 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:41.860305 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:41.860418 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:41.863052 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:41.863074 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:41.863079 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:41.863132 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:41.864589 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:41.865214 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:41.865216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:41.865218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:41.865290 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:41.865700 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:41.865705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:41.865708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:41.865956 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:41.866064 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:41.866515 (+ 451us) server_negotiation.cc:299] Negotiation successful
0806 04:39:41.866654 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:39:41.867532 3975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:41.854960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:37915 (local address 127.31.166.129:53643)
0806 04:39:41.855506 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:41.855562 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:41.856955 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:41.857435 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:41.857441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:41.857793 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:41.858249 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:41.858261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:41.859021 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:41.859024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:41.859967 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:41.859977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:41.860481 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:41.860484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:41.860617 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:41.861156 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:41.861171 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:41.862879 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:41.864715 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:41.864724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:41.864742 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:41.865117 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:41.865382 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:41.865388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:41.865393 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:41.865588 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:41.866064 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:41.866069 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:41.866277 (+ 208us) client_negotiation.cc:769] Sending connection context
0806 04:39:41.866433 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 04:39:41.866668 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":177,"threads_started":1}
I20260806 04:39:41.868785 3973 heartbeater.cc:347] Connected to a master server at 127.31.166.190:37915
I20260806 04:39:41.869045 3973 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:41.869652 3973 heartbeater.cc:511] Master 127.31.166.190:37915 requested a full tablet report, sending...
I20260806 04:39:41.871404 3771 ts_manager.cc:194] Registered new tserver with Master: b6af462da508473aa80b99f8b3434b8a (127.31.166.129:42525)
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
I20260806 04:39:41.873093 3771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:53643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:39:41.892287 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:37915
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:42.002223 3978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:42.002494 3978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:42.002538 3978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:42.006567 3978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:42.006626 3978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:42.006685 3978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:39:42.011972 3978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:37915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:42.013079 3978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:42.014092 3978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:42.023852 3983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:42.023813 3986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:42.023836 3984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:42.024009 3978 server_base.cc:1034] running on GCE node
I20260806 04:39:42.024374 3978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:42.024957 3978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:42.026156 3978 hybrid_clock.cc:648] HybridClock initialized: now 1785991182026128 us; error 52 us; skew 500 ppm
Aug 06 04:39:42 dist-test-slave-vkjp krb5kdc[3723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991182, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:42.028929 3978 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:39:42.029738 3978 webserver.cc:460] Webserver started at http://127.31.166.130:39207/ using document root <none> and password file <none>
I20260806 04:39:42.030295 3978 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:42.030385 3978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:42.030607 3978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:42.032636 3978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "1b87dfdb7c9642af877adf8665917c18"
format_stamp: "Formatted at 2026-08-06 04:39:42 on dist-test-slave-vkjp"
server_key: "7bb68b43922548392afca18bd48fad0a"
server_key_iv: "c6c215e94f030e3f75e7f84c8fee966f"
server_key_version: "encryptionkey@0"
I20260806 04:39:42.033203 3978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "1b87dfdb7c9642af877adf8665917c18"
format_stamp: "Formatted at 2026-08-06 04:39:42 on dist-test-slave-vkjp"
server_key: "7bb68b43922548392afca18bd48fad0a"
server_key_iv: "c6c215e94f030e3f75e7f84c8fee966f"
server_key_version: "encryptionkey@0"
I20260806 04:39:42.037067 3978 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:39:42.039547 3994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:42.040755 3978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:42.040899 3978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "1b87dfdb7c9642af877adf8665917c18"
format_stamp: "Formatted at 2026-08-06 04:39:42 on dist-test-slave-vkjp"
server_key: "7bb68b43922548392afca18bd48fad0a"
server_key_iv: "c6c215e94f030e3f75e7f84c8fee966f"
server_key_version: "encryptionkey@0"
I20260806 04:39:42.041035 3978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:42.057296 3978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:42.060683 3978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:42.060902 3978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:42.061605 3978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:42.062425 3978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:42.062474 3978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:42.062541 3978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:42.062574 3978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:42.072723 3978 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:45255
I20260806 04:39:42.072799 4116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:45255 every 8 connection(s)
I20260806 04:39:42.073802 3978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:39:42.078670 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 3978
I20260806 04:39:42.078832 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:39:42.079224 32410 external_mini_cluster.cc:1428] Setting key 519ca169b80f621300d68ba1fea58720
Aug 06 04:39:42 dist-test-slave-vkjp krb5kdc[3723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991182, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:39:42.088112 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:42.075823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:57673 (local address 127.31.166.190:37915)
0806 04:39:42.075964 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:42.075969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:42.076752 (+ 783us) server_negotiation.cc:407] Connection header received
0806 04:39:42.078208 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:42.078212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:42.078283 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:42.078386 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:42.079761 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:42.080302 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:42.081592 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:42.081815 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:42.081934 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:42.084389 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:42.084392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:42.084394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:42.084419 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:42.086010 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:42.086517 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:42.086522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:42.086523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:42.086605 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:42.086950 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:42.086955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:42.086956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:42.087205 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:42.087341 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:42.087830 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 04:39:42.087952 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:39:42.088928 4119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:42.076099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:37915 (local address 127.31.166.130:57673)
0806 04:39:42.076592 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:42.076642 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:42.077996 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:42.078583 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:42.078589 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:42.079048 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:42.079618 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:42.079631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:42.080429 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:42.080433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:42.081465 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:42.081475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:42.081970 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:42.081975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:42.082148 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:42.082764 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:42.082782 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:42.084220 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:42.086132 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:42.086138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:42.086152 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:42.086412 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:42.086696 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:42.086700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:42.086702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:42.086848 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:42.087336 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:42.087342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:42.087593 (+ 251us) client_negotiation.cc:769] Sending connection context
0806 04:39:42.087790 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 04:39:42.088108 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":129,"threads_started":1}
I20260806 04:39:42.090153 4117 heartbeater.cc:347] Connected to a master server at 127.31.166.190:37915
I20260806 04:39:42.090427 4117 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:42.091161 4117 heartbeater.cc:511] Master 127.31.166.190:37915 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
I20260806 04:39:42.092372 3771 ts_manager.cc:194] Registered new tserver with Master: 1b87dfdb7c9642af877adf8665917c18 (127.31.166.130:45255)
I20260806 04:39:42.093251 3771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:57673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:39:42.108110 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:37915
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:44299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:42.217916 4122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:42.218202 4122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:42.218276 4122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:42.222430 4122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:42.222524 4122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:42.222638 4122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:39:42.228188 4122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:44299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:37915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:42.229310 4122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:42.230332 4122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:42.240165 4130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:42.240208 4127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:42.240499 4122 server_base.cc:1034] running on GCE node
W20260806 04:39:42.240788 4128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:42.241410 4122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:42.242069 4122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:42.243220 4122 hybrid_clock.cc:648] HybridClock initialized: now 1785991182243198 us; error 31 us; skew 500 ppm
Aug 06 04:39:42 dist-test-slave-vkjp krb5kdc[3723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991182, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:42.246384 4122 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:39:42.247367 4122 webserver.cc:460] Webserver started at http://127.31.166.131:45289/ using document root <none> and password file <none>
I20260806 04:39:42.247941 4122 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:42.248003 4122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:42.248245 4122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:42.250445 4122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "760c8f2c4feb49c1b8fe0fa96936bb60"
format_stamp: "Formatted at 2026-08-06 04:39:42 on dist-test-slave-vkjp"
server_key: "8ab5e0e268d85dd385b4ef8f3c7ba4a6"
server_key_iv: "fafd2ab245e8c9974fafa3f7b48962f0"
server_key_version: "encryptionkey@0"
I20260806 04:39:42.250999 4122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "760c8f2c4feb49c1b8fe0fa96936bb60"
format_stamp: "Formatted at 2026-08-06 04:39:42 on dist-test-slave-vkjp"
server_key: "8ab5e0e268d85dd385b4ef8f3c7ba4a6"
server_key_iv: "fafd2ab245e8c9974fafa3f7b48962f0"
server_key_version: "encryptionkey@0"
I20260806 04:39:42.255013 4122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:39:42.257489 4137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:42.258723 4122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 04:39:42.258863 4122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "760c8f2c4feb49c1b8fe0fa96936bb60"
format_stamp: "Formatted at 2026-08-06 04:39:42 on dist-test-slave-vkjp"
server_key: "8ab5e0e268d85dd385b4ef8f3c7ba4a6"
server_key_iv: "fafd2ab245e8c9974fafa3f7b48962f0"
server_key_version: "encryptionkey@0"
I20260806 04:39:42.259028 4122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:42.276175 4122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:42.279810 4122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:42.280043 4122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:42.280679 4122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:42.281565 4122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:42.281636 4122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:42.281713 4122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:42.281754 4122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:42.291307 4122 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:41637
I20260806 04:39:42.291385 4258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:41637 every 8 connection(s)
I20260806 04:39:42.292384 4122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:39:42.294657 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 4122
I20260806 04:39:42.294785 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:39:42.295073 32410 external_mini_cluster.cc:1428] Setting key a09fcac842f277f9af9ec5a516518e8c
Aug 06 04:39:42 dist-test-slave-vkjp krb5kdc[3723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991182, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:42.306264 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:42.294214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:48133 (local address 127.31.166.190:37915)
0806 04:39:42.294374 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:42.294380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:42.295373 (+ 993us) server_negotiation.cc:407] Connection header received
0806 04:39:42.296734 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:42.296737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:42.296788 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:42.296877 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:42.298105 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:42.298630 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:42.299785 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:42.299996 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:42.300077 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:42.302520 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:42.302524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:42.302526 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:42.302555 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:42.304261 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:42.304850 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:42.304853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:42.304854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:42.304940 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:42.305258 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:42.305260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:42.305261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:42.305448 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:42.305524 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:42.306038 (+ 514us) server_negotiation.cc:299] Negotiation successful
0806 04:39:42.306141 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 04:39:42.307238 4261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:42.294647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:37915 (local address 127.31.166.131:48133)
0806 04:39:42.295194 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:42.295250 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:42.296549 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:42.297024 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:42.297030 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:42.297491 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:42.297980 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:42.297993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:42.298735 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:42.298737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:42.299682 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:42.299691 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:42.300124 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:42.300126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:42.300260 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:42.300876 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:42.300891 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:42.302355 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:42.304418 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:42.304423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:42.304436 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:42.304728 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:42.305031 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:42.305034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:42.305035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:42.305173 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:42.305541 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:42.305546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:42.305783 (+ 237us) client_negotiation.cc:769] Sending connection context
0806 04:39:42.305966 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 04:39:42.306238 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":163,"threads_started":1}
I20260806 04:39:42.308584 4259 heartbeater.cc:347] Connected to a master server at 127.31.166.190:37915
I20260806 04:39:42.308851 4259 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:42.309355 4259 heartbeater.cc:511] Master 127.31.166.190:37915 requested a full tablet report, sending...
I20260806 04:39:42.310329 3770 ts_manager.cc:194] Registered new tserver with Master: 760c8f2c4feb49c1b8fe0fa96936bb60 (127.31.166.131:41637)
I20260806 04:39:42.310901 3770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:48133
I20260806 04:39:42.321158 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 04:39:42 dist-test-slave-vkjp krb5kdc[3723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 04:39:42 dist-test-slave-vkjp krb5kdc[3723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.129@KRBTEST.COM
I20260806 04:39:42.348575 4266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:42.338075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.31.166.129:42525)
0806 04:39:42.338436 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:42.338451 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:42.338477 (+ 26us) server_negotiation.cc:407] Connection header received
0806 04:39:42.338528 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:42.338533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:42.338720 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:42.338842 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:42.339790 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:42.340566 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:42.341546 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:42.341927 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:42.342014 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:42.344700 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:42.344711 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:42.344725 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:42.344753 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:42.346557 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:42.347047 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:42.347054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:42.347055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:42.347192 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:42.347485 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:42.347489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:42.347490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:42.347760 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:42.348002 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:42.348208 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 04:39:42.348381 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":198,"threads_started":1}
Aug 06 04:39:42 dist-test-slave-vkjp krb5kdc[3723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:42.360395 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:42.351860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.31.166.190:37915)
0806 04:39:42.352002 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:42.352006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:42.352106 (+ 100us) server_negotiation.cc:407] Connection header received
0806 04:39:42.352228 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:42.352230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:42.352271 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:42.352337 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:42.352995 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:42.353497 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:42.354467 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:42.354669 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:42.354742 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:42.357394 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:42.357398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:42.357400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:42.357429 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:42.358842 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:42.359299 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:42.359302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:42.359303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:42.359385 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:42.359608 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:42.359610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:42.359611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:42.359864 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:42.359976 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:42.360153 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 04:39:42.360256 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 04:39:42.360888 3770 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50028
I20260806 04:39:42.361654 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 3837
I20260806 04:39:42.369755 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 3978
I20260806 04:39:42.376530 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 4122
I20260806 04:39:42.383438 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 3737
2026-08-06T04:39:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2216 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:39:42 dist-test-slave-vkjp krb5kdc[4272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:39:42 dist-test-slave-vkjp krb5kdc[4272](info): set up 2 sockets
Aug 06 04:39:42 dist-test-slave-vkjp krb5kdc[4272](info): commencing operation
krb5kdc: starting...
W20260806 04:39:43.496124 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260806 04:39:43.747594 32410 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.179s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260806 04:39:43.999967 32410 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.252s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/test-user.keytab.
W20260806 04:39:44.445267 32410 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.445s user 0.001s sys 0.001s
Aug 06 04:39:44 dist-test-slave-vkjp krb5kdc[4272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:39:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
W20260806 04:39:45.247951 32410 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.31.166.190: real 0.780s user 0.000s sys 0.003s
I20260806 04:39:45.248565 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:39083
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:39083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:45.360080 4286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:45.360347 4286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:45.360395 4286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:45.364686 4286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:45.364776 4286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:45.364799 4286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:45.364835 4286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:45.364861 4286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:39:45.370316 4286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:39083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:39083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:45.371326 4286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:45.372268 4286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:45.380988 4296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:45.381052 4293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:45.381072 4294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:45.381281 4286 server_base.cc:1034] running on GCE node
I20260806 04:39:45.381883 4286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:45.382992 4286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:45.384200 4286 hybrid_clock.cc:648] HybridClock initialized: now 1785991185384178 us; error 32 us; skew 500 ppm
Aug 06 04:39:45 dist-test-slave-vkjp krb5kdc[4272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991185, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:45.392678 4286 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:39:45.393463 4286 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:45.393569 4286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:45.393821 4286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:45.396080 4286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "a6bd1e270cbb4225a51558f826744702"
format_stamp: "Formatted at 2026-08-06 04:39:45 on dist-test-slave-vkjp"
server_key: "caf29bd880a0f61e51a2cd33006eb824"
server_key_iv: "eb0a53bc1c84f8f1bbb4c627c97eaff5"
server_key_version: "encryptionkey@0"
I20260806 04:39:45.396665 4286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "a6bd1e270cbb4225a51558f826744702"
format_stamp: "Formatted at 2026-08-06 04:39:45 on dist-test-slave-vkjp"
server_key: "caf29bd880a0f61e51a2cd33006eb824"
server_key_iv: "eb0a53bc1c84f8f1bbb4c627c97eaff5"
server_key_version: "encryptionkey@0"
I20260806 04:39:45.400880 4286 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:39:45.403388 4302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:45.404618 4286 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:39:45.404767 4286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "a6bd1e270cbb4225a51558f826744702"
format_stamp: "Formatted at 2026-08-06 04:39:45 on dist-test-slave-vkjp"
server_key: "caf29bd880a0f61e51a2cd33006eb824"
server_key_iv: "eb0a53bc1c84f8f1bbb4c627c97eaff5"
server_key_version: "encryptionkey@0"
I20260806 04:39:45.404888 4286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:45.427579 4286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:45.430660 4286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:45.430877 4286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:45.438148 4286 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:39083
I20260806 04:39:45.438488 4358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:39083 every 8 connection(s)
I20260806 04:39:45.438862 4286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:45.441550 4359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:45.445328 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 4286
I20260806 04:39:45.445489 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:45.445761 32410 external_mini_cluster.cc:1428] Setting key e0d8b1f2aa8adc347b88e7192a44920e
I20260806 04:39:45.447418 4359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702: Bootstrap starting.
I20260806 04:39:45.449918 4359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:45.450660 4359 log.cc:824] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:45.452836 4359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702: No bootstrap required, opened a new log
Aug 06 04:39:45 dist-test-slave-vkjp krb5kdc[4272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:45.457023 4359 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bd1e270cbb4225a51558f826744702" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39083 } }
I20260806 04:39:45.457261 4359 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:45.457345 4359 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6bd1e270cbb4225a51558f826744702, State: Initialized, Role: FOLLOWER
I20260806 04:39:45.457801 4359 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bd1e270cbb4225a51558f826744702" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39083 } }
I20260806 04:39:45.457953 4359 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:45.458038 4359 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:45.458140 4359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:45.459030 4359 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bd1e270cbb4225a51558f826744702" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39083 } }
I20260806 04:39:45.459394 4359 leader_election.cc:302] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6bd1e270cbb4225a51558f826744702; no voters:
I20260806 04:39:45.459725 4359 leader_election.cc:288] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:45.459798 4364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:45.460004 4364 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [term 1 LEADER]: Becoming Leader. State: Replica: a6bd1e270cbb4225a51558f826744702, State: Running, Role: LEADER
I20260806 04:39:45.460366 4364 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bd1e270cbb4225a51558f826744702" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39083 } }
I20260806 04:39:45.460402 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:45.447146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.31.166.190:39083)
0806 04:39:45.447567 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:45.447591 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:45.447628 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:39:45.448248 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:45.448263 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:45.448576 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:45.448850 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:45.449781 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:45.450896 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:45.451992 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:45.452452 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:45.452568 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:45.455342 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:45.455347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:45.455360 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:45.455393 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:45.457089 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:45.457586 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:45.457591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:45.457595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:45.457701 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:45.458074 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:45.458077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:45.458078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:45.458538 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:45.458763 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:45.459060 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 04:39:45.459412 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":150,"threads_started":1}
I20260806 04:39:45.461050 4359 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:45.463660 4365 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6bd1e270cbb4225a51558f826744702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bd1e270cbb4225a51558f826744702" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39083 } } }
I20260806 04:39:45.463790 4365 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:45.464576 4380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:45.464438 4366 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6bd1e270cbb4225a51558f826744702. Latest consensus state: current_term: 1 leader_uuid: "a6bd1e270cbb4225a51558f826744702" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bd1e270cbb4225a51558f826744702" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39083 } } }
I20260806 04:39:45.464893 4366 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:45.467046 4380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:45.472187 4380 catalog_manager.cc:1347] Generated new cluster ID: 85738fdd58be417180247e629fed4d25
I20260806 04:39:45.472259 4380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:45.487044 4380 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:45.488329 4380 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:45.492544 4380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702: Generated new TSK 0
I20260806 04:39:45.493243 4380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:39:45.523142 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39083
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:45.638185 4388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:45.638451 4388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:45.638499 4388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:45.642500 4388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:45.642558 4388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:45.642675 4388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:39:45.647893 4388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:45.648981 4388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:45.649998 4388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:45.659339 4397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:45.659319 4395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:45.659500 4388 server_base.cc:1034] running on GCE node
W20260806 04:39:45.659322 4394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:45.660060 4388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:45.660643 4388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:45.661844 4388 hybrid_clock.cc:648] HybridClock initialized: now 1785991185661805 us; error 50 us; skew 500 ppm
Aug 06 04:39:45 dist-test-slave-vkjp krb5kdc[4272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991185, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:45.665057 4388 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:39:45.665789 4388 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:45.665887 4388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:45.666115 4388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:45.668170 4388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "0c3149cfe8f440608b945777f85d7690"
format_stamp: "Formatted at 2026-08-06 04:39:45 on dist-test-slave-vkjp"
server_key: "7669499973f426edf315d052f40dd00a"
server_key_iv: "c76caaedc1c85e2eedcc6e665f9a428c"
server_key_version: "encryptionkey@0"
I20260806 04:39:45.668728 4388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "0c3149cfe8f440608b945777f85d7690"
format_stamp: "Formatted at 2026-08-06 04:39:45 on dist-test-slave-vkjp"
server_key: "7669499973f426edf315d052f40dd00a"
server_key_iv: "c76caaedc1c85e2eedcc6e665f9a428c"
server_key_version: "encryptionkey@0"
I20260806 04:39:45.672403 4388 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:39:45.674994 4403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:45.676374 4388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:45.676522 4388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "0c3149cfe8f440608b945777f85d7690"
format_stamp: "Formatted at 2026-08-06 04:39:45 on dist-test-slave-vkjp"
server_key: "7669499973f426edf315d052f40dd00a"
server_key_iv: "c76caaedc1c85e2eedcc6e665f9a428c"
server_key_version: "encryptionkey@0"
I20260806 04:39:45.676641 4388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:45.701781 4388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:45.705569 4388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:45.705819 4388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:45.706576 4388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:45.707592 4388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:45.707669 4388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:45.707751 4388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:45.707805 4388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:45.717907 4388 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:40051
I20260806 04:39:45.717928 4521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:40051 every 8 connection(s)
I20260806 04:39:45.718570 4388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:39:45.719851 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 4388
I20260806 04:39:45.719988 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:39:45.720273 32410 external_mini_cluster.cc:1428] Setting key 5c4363b359de0cc7d93ffa78de27fa20
Aug 06 04:39:45 dist-test-slave-vkjp krb5kdc[4272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991185, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
I20260806 04:39:45.734484 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:45.720653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:52077 (local address 127.31.166.190:39083)
0806 04:39:45.720790 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:45.720794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:45.721998 (+ 1204us) server_negotiation.cc:407] Connection header received
0806 04:39:45.724098 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:45.724105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:45.724172 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:45.724275 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:45.725469 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:45.726256 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:45.727575 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:45.727824 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:45.727938 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:45.730461 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:45.730464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:45.730466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:45.730490 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:45.732162 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:45.732678 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:45.732681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:45.732683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:45.732782 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:45.733231 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:45.733234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:45.733235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:45.733455 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:45.733548 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:45.734178 (+ 630us) server_negotiation.cc:299] Negotiation successful
0806 04:39:45.734343 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 04:39:45.735677 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:45.721195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39083 (local address 127.31.166.129:52077)
0806 04:39:45.721799 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:45.721894 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:45.723817 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:45.724453 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:45.724461 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:45.724813 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:45.725312 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:45.725325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:45.726459 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:45.726462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:45.727450 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:45.727459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:45.727980 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:45.727984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:45.728153 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:45.728820 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:45.728837 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:45.730262 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:45.732274 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:45.732279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:45.732290 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:45.732559 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:45.732890 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:45.732892 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:45.732894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:45.733032 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:45.733564 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:45.733569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:45.733875 (+ 306us) client_negotiation.cc:769] Sending connection context
0806 04:39:45.734085 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 04:39:45.734405 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":164,"threads_started":1}
I20260806 04:39:45.737143 4522 heartbeater.cc:347] Connected to a master server at 127.31.166.190:39083
I20260806 04:39:45.737389 4522 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:45.737838 4522 heartbeater.cc:511] Master 127.31.166.190:39083 requested a full tablet report, sending...
I20260806 04:39:45.739394 4322 ts_manager.cc:194] Registered new tserver with Master: 0c3149cfe8f440608b945777f85d7690 (127.31.166.129:40051)
I20260806 04:39:45.740518 4322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:52077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:39:45.750600 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39083
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:45.870718 4527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:45.870997 4527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:45.871075 4527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:45.875130 4527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:45.875218 4527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:45.875308 4527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:39:45.880571 4527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:45.881603 4527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:45.882656 4527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:45.891741 4535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:45.891716 4532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:45.891897 4533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:45.891928 4527 server_base.cc:1034] running on GCE node
I20260806 04:39:45.892458 4527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:45.892994 4527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:45.894151 4527 hybrid_clock.cc:648] HybridClock initialized: now 1785991185894117 us; error 43 us; skew 500 ppm
Aug 06 04:39:45 dist-test-slave-vkjp krb5kdc[4272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991185, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:45.896903 4527 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:39:45.897542 4527 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:45.897603 4527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:45.897758 4527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:45.899770 4527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "858d4a54e46546108301ea0ef2f12ba2"
format_stamp: "Formatted at 2026-08-06 04:39:45 on dist-test-slave-vkjp"
server_key: "abd504a2d52d73dffb3dadb4641bb4ae"
server_key_iv: "f2dc9d01d815e9a9c1ef7dc500f29c73"
server_key_version: "encryptionkey@0"
I20260806 04:39:45.900271 4527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "858d4a54e46546108301ea0ef2f12ba2"
format_stamp: "Formatted at 2026-08-06 04:39:45 on dist-test-slave-vkjp"
server_key: "abd504a2d52d73dffb3dadb4641bb4ae"
server_key_iv: "f2dc9d01d815e9a9c1ef7dc500f29c73"
server_key_version: "encryptionkey@0"
I20260806 04:39:45.903910 4527 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:39:45.906240 4541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:45.907586 4527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:45.907727 4527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "858d4a54e46546108301ea0ef2f12ba2"
format_stamp: "Formatted at 2026-08-06 04:39:45 on dist-test-slave-vkjp"
server_key: "abd504a2d52d73dffb3dadb4641bb4ae"
server_key_iv: "f2dc9d01d815e9a9c1ef7dc500f29c73"
server_key_version: "encryptionkey@0"
I20260806 04:39:45.907840 4527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:45.934693 4527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:45.937687 4527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:45.937897 4527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:45.938479 4527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:45.939368 4527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:45.939440 4527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:45.939512 4527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:45.939565 4527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:45.948491 4527 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:36521
I20260806 04:39:45.948508 4662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:36521 every 8 connection(s)
I20260806 04:39:45.949098 4527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:39:45.957119 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 4527
I20260806 04:39:45.957261 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:39:45.957584 32410 external_mini_cluster.cc:1428] Setting key 81ff2e88ff0759f5d117879e4e319e84
Aug 06 04:39:45 dist-test-slave-vkjp krb5kdc[4272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991185, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:45.963579 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:45.950813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:49779 (local address 127.31.166.190:39083)
0806 04:39:45.950959 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:45.950963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:45.951859 (+ 896us) server_negotiation.cc:407] Connection header received
0806 04:39:45.953170 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:45.953173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:45.953232 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:45.953358 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:45.954494 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:45.955038 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:45.956236 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:45.956472 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:45.956587 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:45.959458 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:45.959463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:45.959466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:45.959493 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:45.961250 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:45.961764 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:45.961767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:45.961769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:45.961897 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:45.962275 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:45.962278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:45.962280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:45.962513 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:45.962632 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:45.963255 (+ 623us) server_negotiation.cc:299] Negotiation successful
0806 04:39:45.963395 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 04:39:45.964351 4665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:45.951201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39083 (local address 127.31.166.130:49779)
0806 04:39:45.951718 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:45.951760 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:45.952966 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:45.953538 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:45.953545 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:45.953888 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:45.954339 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:45.954350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:45.955220 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:45.955223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:45.956106 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:45.956116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:45.956627 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:45.956631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:45.956795 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:45.957476 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:45.957491 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:45.959257 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:45.961373 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:45.961379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:45.961393 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:45.961649 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:45.961988 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:45.961993 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:45.961995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:45.962136 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:45.962632 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:45.962639 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:45.962939 (+ 300us) client_negotiation.cc:769] Sending connection context
0806 04:39:45.963195 (+ 256us) client_negotiation.cc:240] Negotiation successful
0806 04:39:45.963533 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":124,"threads_started":1}
I20260806 04:39:45.965499 4663 heartbeater.cc:347] Connected to a master server at 127.31.166.190:39083
I20260806 04:39:45.965709 4663 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:45.966105 4663 heartbeater.cc:511] Master 127.31.166.190:39083 requested a full tablet report, sending...
I20260806 04:39:45.967175 4322 ts_manager.cc:194] Registered new tserver with Master: 858d4a54e46546108301ea0ef2f12ba2 (127.31.166.130:36521)
I20260806 04:39:45.967742 4322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:49779
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:39:45.986582 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39083
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:38523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:46.101166 4668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:46.101432 4668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:46.101480 4668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:46.105626 4668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:46.105700 4668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:46.105799 4668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:39:46.111340 4668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:38523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:46.112476 4668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:46.113539 4668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:46.123066 4676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:46.123066 4673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:46.123138 4668 server_base.cc:1034] running on GCE node
W20260806 04:39:46.123387 4674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:46.123850 4668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:46.124482 4668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:46.125672 4668 hybrid_clock.cc:648] HybridClock initialized: now 1785991186125663 us; error 29 us; skew 500 ppm
Aug 06 04:39:46 dist-test-slave-vkjp krb5kdc[4272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991186, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:46.128432 4668 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:39:46.129179 4668 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:46.129287 4668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:46.129521 4668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:46.131623 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "9f486b786c1c46438e5411a59a29a949"
format_stamp: "Formatted at 2026-08-06 04:39:46 on dist-test-slave-vkjp"
server_key: "7f9e0fab197bc34c8f168d8e929f81e2"
server_key_iv: "1356dd2607e518a794a40046783aa1f2"
server_key_version: "encryptionkey@0"
I20260806 04:39:46.132191 4668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "9f486b786c1c46438e5411a59a29a949"
format_stamp: "Formatted at 2026-08-06 04:39:46 on dist-test-slave-vkjp"
server_key: "7f9e0fab197bc34c8f168d8e929f81e2"
server_key_iv: "1356dd2607e518a794a40046783aa1f2"
server_key_version: "encryptionkey@0"
I20260806 04:39:46.135907 4668 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:39:46.138460 4682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:46.139871 4668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:46.140015 4668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "9f486b786c1c46438e5411a59a29a949"
format_stamp: "Formatted at 2026-08-06 04:39:46 on dist-test-slave-vkjp"
server_key: "7f9e0fab197bc34c8f168d8e929f81e2"
server_key_iv: "1356dd2607e518a794a40046783aa1f2"
server_key_version: "encryptionkey@0"
I20260806 04:39:46.140134 4668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:46.150699 4668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:46.153865 4668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:46.154083 4668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:46.154675 4668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:46.155628 4668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:46.155701 4668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:46.155776 4668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:46.155812 4668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:46.165035 4668 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:39479
I20260806 04:39:46.165046 4801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:39479 every 8 connection(s)
I20260806 04:39:46.165627 4668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:39:46.172832 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 4668
I20260806 04:39:46.172976 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:39:46.173274 32410 external_mini_cluster.cc:1428] Setting key 55b425813351e966a53ca7a4b8b5abc8
Aug 06 04:39:46 dist-test-slave-vkjp krb5kdc[4272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991186, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:46.179611 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.167371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:57813 (local address 127.31.166.190:39083)
0806 04:39:46.167521 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:46.167524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:46.168503 (+ 979us) server_negotiation.cc:407] Connection header received
0806 04:39:46.169903 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:46.169907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:46.169964 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:46.170068 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:46.171211 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.171741 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.172901 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.173224 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.173336 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:46.176042 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:46.176045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:46.176048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:46.176075 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:46.177516 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:46.178053 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:46.178056 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:46.178058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:46.178149 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:46.178525 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:46.178530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:46.178532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:46.178715 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:46.178830 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:46.179331 (+ 501us) server_negotiation.cc:299] Negotiation successful
0806 04:39:46.179449 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:39:46.180444 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.167814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39083 (local address 127.31.166.131:57813)
0806 04:39:46.168337 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:46.168382 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:46.169693 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:46.170234 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:46.170240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:46.170590 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:46.171041 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:46.171053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.171871 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.171874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:46.172770 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:46.172779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.173378 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.173382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:46.173546 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:46.174224 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:46.174240 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:46.175886 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:46.177646 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:46.177652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:46.177666 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:46.177943 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:46.178242 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:46.178246 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:46.178248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:46.178426 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:46.178820 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:46.178826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:46.179056 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 04:39:46.179290 (+ 234us) client_negotiation.cc:240] Negotiation successful
0806 04:39:46.179640 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":114,"threads_started":1}
I20260806 04:39:46.181524 4802 heartbeater.cc:347] Connected to a master server at 127.31.166.190:39083
I20260806 04:39:46.181804 4802 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:46.182194 4802 heartbeater.cc:511] Master 127.31.166.190:39083 requested a full tablet report, sending...
I20260806 04:39:46.183248 4322 ts_manager.cc:194] Registered new tserver with Master: 9f486b786c1c46438e5411a59a29a949 (127.31.166.131:39479)
I20260806 04:39:46.183827 4322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:57813
I20260806 04:39:46.187233 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:39:46.196333 4362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.189099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.31.166.190:39083)
0806 04:39:46.189243 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:46.189248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:46.189465 (+ 217us) server_negotiation.cc:407] Connection header received
0806 04:39:46.189613 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:46.189616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:46.189659 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:46.189744 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:46.190432 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.190939 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.191845 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.192076 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.192185 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:46.193450 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:46.193454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:46.193457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:46.193481 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:46.194622 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:46.195052 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:46.195056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:46.195057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:46.195177 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:46.195460 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:46.195463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:46.195465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:46.195679 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:46.195823 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:46.196059 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 04:39:46.196173 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:39:46.198915 4322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:39:46.201874 4322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:39:46.216035 4815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.210441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:40051 (local address 127.0.0.1:52852)
0806 04:39:46.211483 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:46.211496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:46.211593 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:46.212201 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:46.212204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:46.212219 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:46.212326 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:46.212332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.213746 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.213749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:46.214617 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:46.214626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.215480 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.215483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:46.215598 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:46.215638 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 04:39:46.215772 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 04:39:46.215889 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"thread_start_us":86,"threads_started":1}
I20260806 04:39:46.216046 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.210808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.31.166.129:40051)
0806 04:39:46.211252 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:46.211258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:46.211609 (+ 351us) server_negotiation.cc:407] Connection header received
0806 04:39:46.211695 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:46.211701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:46.211908 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:46.212047 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:46.212449 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.213609 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.214743 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.215358 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.215492 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:46.215620 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:46.215805 (+ 185us) server_negotiation.cc:299] Negotiation successful
0806 04:39:46.215904 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":134,"threads_started":1}
I20260806 04:39:46.216670 4814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.210022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:39479 (local address 127.0.0.1:58086)
0806 04:39:46.210857 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:46.210907 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:46.211032 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:46.212059 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:46.212062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:46.212079 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:46.212193 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:46.212200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.213517 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.213520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:46.214579 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:46.214589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.216271 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.216275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:46.216365 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:46.216397 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 04:39:46.216454 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 04:39:46.216542 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":92,"threads_started":1}
I20260806 04:39:46.217540 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.210202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58086 (local address 127.31.166.131:39479)
0806 04:39:46.211661 (+ 1459us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:46.211672 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:46.211690 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:39:46.211751 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:46.211755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:46.211900 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:46.212334 (+ 434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:46.212510 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.213348 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.215220 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.216000 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.217149 (+ 1149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:46.217268 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:46.217314 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 04:39:46.217398 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1334,"thread_start_us":211,"threads_started":1}
I20260806 04:39:46.217583 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.210793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.31.166.130:36521)
0806 04:39:46.211246 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:46.211262 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:46.211427 (+ 165us) server_negotiation.cc:407] Connection header received
0806 04:39:46.211517 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:46.211526 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:46.211730 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:46.211935 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:46.212183 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.213351 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.215612 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.216427 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.217162 (+ 735us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:46.217297 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:46.217400 (+ 103us) server_negotiation.cc:299] Negotiation successful
0806 04:39:46.217470 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":99,"threads_started":1}
I20260806 04:39:46.217831 4817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.210201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:36521 (local address 127.0.0.1:46734)
0806 04:39:46.211311 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:46.211323 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:46.211414 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:46.211933 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:46.211939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:46.211960 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:46.212087 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:46.212093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.213497 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.213500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:46.215477 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:46.215487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.216555 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.216558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:46.216642 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:46.216662 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 04:39:46.216711 (+ 49us) client_negotiation.cc:240] Negotiation successful
0806 04:39:46.216793 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":978,"thread_start_us":99,"threads_started":1}
I20260806 04:39:46.219130 4452 tablet_service.cc:1467] Processing CreateTablet for tablet d71e038294894fef90b7516366c6a152 (DEFAULT_TABLE table=test-table [id=6c07791612ee461aa250521ba0199256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:46.219918 4732 tablet_service.cc:1467] Processing CreateTablet for tablet d71e038294894fef90b7516366c6a152 (DEFAULT_TABLE table=test-table [id=6c07791612ee461aa250521ba0199256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:46.220302 4452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d71e038294894fef90b7516366c6a152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:46.220839 4732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d71e038294894fef90b7516366c6a152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:46.220566 4591 tablet_service.cc:1467] Processing CreateTablet for tablet d71e038294894fef90b7516366c6a152 (DEFAULT_TABLE table=test-table [id=6c07791612ee461aa250521ba0199256]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:46.221508 4591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d71e038294894fef90b7516366c6a152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:46.226404 4820 tablet_bootstrap.cc:492] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949: Bootstrap starting.
I20260806 04:39:46.228197 4821 tablet_bootstrap.cc:492] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690: Bootstrap starting.
I20260806 04:39:46.228683 4820 tablet_bootstrap.cc:654] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:46.229341 4822 tablet_bootstrap.cc:492] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2: Bootstrap starting.
I20260806 04:39:46.230060 4820 log.cc:824] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:46.230257 4821 tablet_bootstrap.cc:654] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:46.231314 4822 tablet_bootstrap.cc:654] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:46.231546 4821 log.cc:824] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:46.232355 4820 tablet_bootstrap.cc:492] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949: No bootstrap required, opened a new log
I20260806 04:39:46.232718 4822 log.cc:824] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:46.232791 4820 ts_tablet_manager.cc:1397] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:39:46.233500 4821 tablet_bootstrap.cc:492] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690: No bootstrap required, opened a new log
I20260806 04:39:46.233704 4821 ts_tablet_manager.cc:1397] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:39:46.234586 4822 tablet_bootstrap.cc:492] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2: No bootstrap required, opened a new log
I20260806 04:39:46.234764 4822 ts_tablet_manager.cc:1397] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:39:46.236675 4821 raft_consensus.cc:348] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f486b786c1c46438e5411a59a29a949" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 39479 } } peers { permanent_uuid: "858d4a54e46546108301ea0ef2f12ba2" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 36521 } } peers { permanent_uuid: "0c3149cfe8f440608b945777f85d7690" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40051 } }
I20260806 04:39:46.236696 4820 raft_consensus.cc:348] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f486b786c1c46438e5411a59a29a949" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 39479 } } peers { permanent_uuid: "858d4a54e46546108301ea0ef2f12ba2" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 36521 } } peers { permanent_uuid: "0c3149cfe8f440608b945777f85d7690" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40051 } }
I20260806 04:39:46.236910 4821 raft_consensus.cc:374] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:46.236949 4820 raft_consensus.cc:374] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:46.236958 4821 raft_consensus.cc:729] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c3149cfe8f440608b945777f85d7690, State: Initialized, Role: FOLLOWER
I20260806 04:39:46.236995 4820 raft_consensus.cc:729] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f486b786c1c46438e5411a59a29a949, State: Initialized, Role: FOLLOWER
I20260806 04:39:46.237403 4821 consensus_queue.cc:260] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f486b786c1c46438e5411a59a29a949" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 39479 } } peers { permanent_uuid: "858d4a54e46546108301ea0ef2f12ba2" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 36521 } } peers { permanent_uuid: "0c3149cfe8f440608b945777f85d7690" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40051 } }
I20260806 04:39:46.237452 4820 consensus_queue.cc:260] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f486b786c1c46438e5411a59a29a949" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 39479 } } peers { permanent_uuid: "858d4a54e46546108301ea0ef2f12ba2" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 36521 } } peers { permanent_uuid: "0c3149cfe8f440608b945777f85d7690" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40051 } }
I20260806 04:39:46.238369 4820 ts_tablet_manager.cc:1428] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260806 04:39:46.238410 4821 ts_tablet_manager.cc:1428] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 04:39:46.238449 4822 raft_consensus.cc:348] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f486b786c1c46438e5411a59a29a949" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 39479 } } peers { permanent_uuid: "858d4a54e46546108301ea0ef2f12ba2" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 36521 } } peers { permanent_uuid: "0c3149cfe8f440608b945777f85d7690" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40051 } }
I20260806 04:39:46.238686 4822 raft_consensus.cc:374] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:46.238740 4822 raft_consensus.cc:729] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 858d4a54e46546108301ea0ef2f12ba2, State: Initialized, Role: FOLLOWER
I20260806 04:39:46.238749 4802 heartbeater.cc:503] Master 127.31.166.190:39083 was elected leader, sending a full tablet report...
I20260806 04:39:46.238749 4522 heartbeater.cc:503] Master 127.31.166.190:39083 was elected leader, sending a full tablet report...
I20260806 04:39:46.239190 4822 consensus_queue.cc:260] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f486b786c1c46438e5411a59a29a949" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 39479 } } peers { permanent_uuid: "858d4a54e46546108301ea0ef2f12ba2" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 36521 } } peers { permanent_uuid: "0c3149cfe8f440608b945777f85d7690" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40051 } }
I20260806 04:39:46.239790 4663 heartbeater.cc:503] Master 127.31.166.190:39083 was elected leader, sending a full tablet report...
I20260806 04:39:46.241124 4822 ts_tablet_manager.cc:1428] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:39:46.261090 4827 raft_consensus.cc:482] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:39:46.261309 4827 raft_consensus.cc:504] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f486b786c1c46438e5411a59a29a949" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 39479 } } peers { permanent_uuid: "858d4a54e46546108301ea0ef2f12ba2" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 36521 } } peers { permanent_uuid: "0c3149cfe8f440608b945777f85d7690" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40051 } }
I20260806 04:39:46.262677 4827 leader_election.cc:288] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 858d4a54e46546108301ea0ef2f12ba2 (127.31.166.130:36521), 0c3149cfe8f440608b945777f85d7690 (127.31.166.129:40051)
I20260806 04:39:46.266875 4818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.263008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:38067 (local address 127.31.166.130:36521)
0806 04:39:46.263173 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:46.263177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:46.263195 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:39:46.263363 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:46.263367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:46.263447 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:46.263522 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:46.263903 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.264698 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.265790 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.266217 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.266407 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:46.266465 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:46.266680 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 04:39:46.266752 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:39:46.266875 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.262898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:36521 (local address 127.31.166.131:38067)
0806 04:39:46.263132 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:46.263148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:46.263242 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:46.263595 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:46.263601 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:46.263618 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:46.263760 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:46.263768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.264824 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.264827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:46.265658 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:46.265667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.266393 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.266397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:46.266523 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:46.266572 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 04:39:46.266662 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 04:39:46.266733 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":25}
I20260806 04:39:46.266983 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.263152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:40835 (local address 127.31.166.129:40051)
0806 04:39:46.263269 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:46.263275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:46.263521 (+ 246us) server_negotiation.cc:407] Connection header received
0806 04:39:46.263730 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:46.263736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:46.263799 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:46.263899 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:46.264160 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.264879 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.265973 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.266370 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.266500 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:46.266564 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:46.266779 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 04:39:46.266856 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:39:46.267007 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.262979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:40051 (local address 127.31.166.131:40835)
0806 04:39:46.263420 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:46.263438 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:46.263560 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:46.263910 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:46.263913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:46.263930 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:46.264053 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:46.264059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.264998 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.265002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:46.265844 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:46.265854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.266495 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.266499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:46.266648 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:46.266666 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 04:39:46.266749 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 04:39:46.266822 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":46,"thread_start_us":127,"threads_started":1}
I20260806 04:39:46.267650 4473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71e038294894fef90b7516366c6a152" candidate_uuid: "9f486b786c1c46438e5411a59a29a949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3149cfe8f440608b945777f85d7690" is_pre_election: true
I20260806 04:39:46.267647 4613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71e038294894fef90b7516366c6a152" candidate_uuid: "9f486b786c1c46438e5411a59a29a949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858d4a54e46546108301ea0ef2f12ba2" is_pre_election: true
I20260806 04:39:46.267961 4613 raft_consensus.cc:2454] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f486b786c1c46438e5411a59a29a949 in term 0.
I20260806 04:39:46.267961 4473 raft_consensus.cc:2454] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f486b786c1c46438e5411a59a29a949 in term 0.
I20260806 04:39:46.268523 4686 leader_election.cc:302] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0c3149cfe8f440608b945777f85d7690, 858d4a54e46546108301ea0ef2f12ba2, 9f486b786c1c46438e5411a59a29a949; no voters:
I20260806 04:39:46.268817 4827 raft_consensus.cc:2781] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:39:46.268898 4827 raft_consensus.cc:482] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:39:46.268929 4827 raft_consensus.cc:3037] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:46.270123 4827 raft_consensus.cc:504] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f486b786c1c46438e5411a59a29a949" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 39479 } } peers { permanent_uuid: "858d4a54e46546108301ea0ef2f12ba2" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 36521 } } peers { permanent_uuid: "0c3149cfe8f440608b945777f85d7690" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40051 } }
I20260806 04:39:46.270469 4827 leader_election.cc:288] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [CANDIDATE]: Term 1 election: Requested vote from peers 858d4a54e46546108301ea0ef2f12ba2 (127.31.166.130:36521), 0c3149cfe8f440608b945777f85d7690 (127.31.166.129:40051)
I20260806 04:39:46.270900 4473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71e038294894fef90b7516366c6a152" candidate_uuid: "9f486b786c1c46438e5411a59a29a949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c3149cfe8f440608b945777f85d7690"
I20260806 04:39:46.270900 4613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71e038294894fef90b7516366c6a152" candidate_uuid: "9f486b786c1c46438e5411a59a29a949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "858d4a54e46546108301ea0ef2f12ba2"
I20260806 04:39:46.271034 4473 raft_consensus.cc:3037] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:46.271034 4613 raft_consensus.cc:3037] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:46.272461 4473 raft_consensus.cc:2454] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f486b786c1c46438e5411a59a29a949 in term 1.
I20260806 04:39:46.272461 4613 raft_consensus.cc:2454] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f486b786c1c46438e5411a59a29a949 in term 1.
I20260806 04:39:46.272782 4683 leader_election.cc:302] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 858d4a54e46546108301ea0ef2f12ba2, 9f486b786c1c46438e5411a59a29a949; no voters:
I20260806 04:39:46.272960 4827 raft_consensus.cc:2781] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:46.273171 4827 raft_consensus.cc:686] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [term 1 LEADER]: Becoming Leader. State: Replica: 9f486b786c1c46438e5411a59a29a949, State: Running, Role: LEADER
I20260806 04:39:46.273494 4827 consensus_queue.cc:237] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f486b786c1c46438e5411a59a29a949" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 39479 } } peers { permanent_uuid: "858d4a54e46546108301ea0ef2f12ba2" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 36521 } } peers { permanent_uuid: "0c3149cfe8f440608b945777f85d7690" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40051 } }
I20260806 04:39:46.276928 4322 catalog_manager.cc:5495] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f486b786c1c46438e5411a59a29a949 (127.31.166.131). New cstate: current_term: 1 leader_uuid: "9f486b786c1c46438e5411a59a29a949" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f486b786c1c46438e5411a59a29a949" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 39479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "858d4a54e46546108301ea0ef2f12ba2" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 36521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c3149cfe8f440608b945777f85d7690" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40051 } health_report { overall_health: UNKNOWN } } }
I20260806 04:39:46.300673 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.297028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.31.166.131:39479)
0806 04:39:46.297177 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:46.297181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:46.297195 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:39:46.297381 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:46.297384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:46.297438 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:46.297550 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:46.297793 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.298301 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.299243 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:46.299459 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:46.299565 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:46.299881 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:46.300268 (+ 387us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:46.300392 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:46.300473 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 04:39:46.300529 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 04:39:46.306624 4473 raft_consensus.cc:1261] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690 [term 1 FOLLOWER]: Refusing update from remote peer 9f486b786c1c46438e5411a59a29a949: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:46.306648 4613 raft_consensus.cc:1261] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2 [term 1 FOLLOWER]: Refusing update from remote peer 9f486b786c1c46438e5411a59a29a949: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:46.307356 4830 consensus_queue.cc:1035] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c3149cfe8f440608b945777f85d7690" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:46.307528 4827 consensus_queue.cc:1035] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [LEADER]: Connected to new peer: Peer: permanent_uuid: "858d4a54e46546108301ea0ef2f12ba2" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:46.323913 4320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:46.324107 4320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:46.326650 4320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6bd1e270cbb4225a51558f826744702: Sending DeleteTablet for 3 replicas of tablet d71e038294894fef90b7516366c6a152
I20260806 04:39:46.327454 4591 tablet_service.cc:1514] Processing DeleteTablet for tablet d71e038294894fef90b7516366c6a152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:46 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:46734
I20260806 04:39:46.327441 4732 tablet_service.cc:1514] Processing DeleteTablet for tablet d71e038294894fef90b7516366c6a152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:46 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:58086
I20260806 04:39:46.327574 4452 tablet_service.cc:1514] Processing DeleteTablet for tablet d71e038294894fef90b7516366c6a152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:46 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:52852
I20260806 04:39:46.328035 4844 tablet_replica.cc:330] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949: stopping tablet replica
I20260806 04:39:46.328102 4846 tablet_replica.cc:330] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690: stopping tablet replica
I20260806 04:39:46.328436 4846 raft_consensus.cc:2229] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:46.328758 4846 raft_consensus.cc:2258] T d71e038294894fef90b7516366c6a152 P 0c3149cfe8f440608b945777f85d7690 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:46.328826 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 4388
I20260806 04:39:46.328821 4844 raft_consensus.cc:2229] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:39:46.332253 4844 raft_consensus.cc:2258] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:46.332763 4845 tablet_replica.cc:330] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2: stopping tablet replica
I20260806 04:39:46.333060 4845 raft_consensus.cc:2229] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:46.333278 4845 raft_consensus.cc:2258] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:46.334537 4845 ts_tablet_manager.cc:1905] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:46.336933 4845 ts_tablet_manager.cc:1918] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:39:46.337005 4845 log.cc:1196] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/d71e038294894fef90b7516366c6a152
I20260806 04:39:46.337244 4845 ts_tablet_manager.cc:1939] T d71e038294894fef90b7516366c6a152 P 858d4a54e46546108301ea0ef2f12ba2: Deleting consensus metadata
I20260806 04:39:46.338443 4303 catalog_manager.cc:4841] TS 858d4a54e46546108301ea0ef2f12ba2 (127.31.166.130:36521): tablet d71e038294894fef90b7516366c6a152 (table test-table [id=6c07791612ee461aa250521ba0199256]) successfully deleted
W20260806 04:39:46.338727 4308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:40051 (error 108)
I20260806 04:39:46.339438 4844 ts_tablet_manager.cc:1905] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:46.339725 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 4527
I20260806 04:39:46.340159 4817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.339847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:40051 (local address 127.0.0.1:52862)
0806 04:39:46.340012 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:46.340076 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:40051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260806 04:39:46.340299 4308 catalog_manager.cc:4584] TS 0c3149cfe8f440608b945777f85d7690 (127.31.166.129:40051): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d71e038294894fef90b7516366c6a152: Network error: Client connection negotiation failed: client connection to 127.31.166.129:40051: connect: Connection refused (error 111)
I20260806 04:39:46.342839 4844 ts_tablet_manager.cc:1918] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:39:46.342938 4844 log.cc:1196] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/d71e038294894fef90b7516366c6a152
I20260806 04:39:46.343297 4844 ts_tablet_manager.cc:1939] T d71e038294894fef90b7516366c6a152 P 9f486b786c1c46438e5411a59a29a949: Deleting consensus metadata
I20260806 04:39:46.344426 4307 catalog_manager.cc:4841] TS 9f486b786c1c46438e5411a59a29a949 (127.31.166.131:39479): tablet d71e038294894fef90b7516366c6a152 (table test-table [id=6c07791612ee461aa250521ba0199256]) successfully deleted
I20260806 04:39:46.347947 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 4668
I20260806 04:39:46.354465 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 4286
2026-08-06T04:39:46Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3970 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T04:39:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:46Z Disabled control of system clock
I20260806 04:39:46.385512 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:36107
--webserver_interface=127.31.166.190
--webserver_port=0
--builtin_ntp_servers=127.31.166.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:36107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 04:39:46.496862 4851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:46.497133 4851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:46.497170 4851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:46.501291 4851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:46.501353 4851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:46.501369 4851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:46.501386 4851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:46.501401 4851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:39:46.506150 4851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:36107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:36107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:46.507280 4851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:46.508616 4851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:46.517021 4860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:46.517042 4858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:46.517037 4857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:46.517241 4851 server_base.cc:1034] running on GCE node
I20260806 04:39:46.517887 4851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:46.518982 4851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:46.520169 4851 hybrid_clock.cc:648] HybridClock initialized: now 1785991186520150 us; error 29 us; skew 500 ppm
I20260806 04:39:46.522102 4851 webserver.cc:460] Webserver started at http://127.31.166.190:40177/ using document root <none> and password file <none>
I20260806 04:39:46.522728 4851 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:46.522794 4851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:46.523015 4851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:46.525243 4851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "a61fc15a84064ba0b1aad915f3207c5f"
format_stamp: "Formatted at 2026-08-06 04:39:46 on dist-test-slave-vkjp"
server_key: "cfef8ab563fb2d37840733255096947d"
server_key_iv: "e244066cf2900668011bd9ef880bb8fd"
server_key_version: "encryptionkey@0"
I20260806 04:39:46.525832 4851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "a61fc15a84064ba0b1aad915f3207c5f"
format_stamp: "Formatted at 2026-08-06 04:39:46 on dist-test-slave-vkjp"
server_key: "cfef8ab563fb2d37840733255096947d"
server_key_iv: "e244066cf2900668011bd9ef880bb8fd"
server_key_version: "encryptionkey@0"
I20260806 04:39:46.529671 4851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:39:46.532126 4866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:46.533349 4851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:46.533483 4851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "a61fc15a84064ba0b1aad915f3207c5f"
format_stamp: "Formatted at 2026-08-06 04:39:46 on dist-test-slave-vkjp"
server_key: "cfef8ab563fb2d37840733255096947d"
server_key_iv: "e244066cf2900668011bd9ef880bb8fd"
server_key_version: "encryptionkey@0"
I20260806 04:39:46.533600 4851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:46.538146 4851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:46.538734 4851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:46.538925 4851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:46.545446 4851 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:36107
I20260806 04:39:46.545454 4921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:36107 every 8 connection(s)
I20260806 04:39:46.546399 4851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:46.549585 4922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:46.551015 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 4851
I20260806 04:39:46.551167 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:46.551501 32410 external_mini_cluster.cc:1428] Setting key e5c5a09f49d1071dae2d190f7abcbe57
I20260806 04:39:46.556092 4922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f: Bootstrap starting.
I20260806 04:39:46.557219 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.552990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.31.166.190:36107)
0806 04:39:46.553650 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:46.553666 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:46.553701 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:39:46.554407 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:46.554425 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:46.554764 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:46.555143 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:46.555981 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:46.555985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:46.555996 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:46.556007 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:46.556061 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:46.556077 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:46.556233 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:46.556309 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 04:39:46.556550 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":143,"threads_started":1}
I20260806 04:39:46.558597 4922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:46.559477 4922 log.cc:824] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:46.561363 4922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f: No bootstrap required, opened a new log
I20260806 04:39:46.564970 4922 raft_consensus.cc:348] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61fc15a84064ba0b1aad915f3207c5f" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36107 } }
I20260806 04:39:46.565207 4922 raft_consensus.cc:374] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:46.565284 4922 raft_consensus.cc:729] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61fc15a84064ba0b1aad915f3207c5f, State: Initialized, Role: FOLLOWER
I20260806 04:39:46.565727 4922 consensus_queue.cc:260] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61fc15a84064ba0b1aad915f3207c5f" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36107 } }
I20260806 04:39:46.565876 4922 raft_consensus.cc:388] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:46.565958 4922 raft_consensus.cc:482] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:46.566056 4922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:46.566953 4922 raft_consensus.cc:504] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61fc15a84064ba0b1aad915f3207c5f" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36107 } }
I20260806 04:39:46.567298 4922 leader_election.cc:302] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a61fc15a84064ba0b1aad915f3207c5f; no voters:
I20260806 04:39:46.567646 4922 leader_election.cc:288] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:46.567719 4927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:46.567919 4927 raft_consensus.cc:686] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [term 1 LEADER]: Becoming Leader. State: Replica: a61fc15a84064ba0b1aad915f3207c5f, State: Running, Role: LEADER
I20260806 04:39:46.568197 4927 consensus_queue.cc:237] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61fc15a84064ba0b1aad915f3207c5f" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36107 } }
I20260806 04:39:46.568935 4922 sys_catalog.cc:564] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:46.569944 4928 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a61fc15a84064ba0b1aad915f3207c5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61fc15a84064ba0b1aad915f3207c5f" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36107 } } }
I20260806 04:39:46.569983 4929 sys_catalog.cc:455] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a61fc15a84064ba0b1aad915f3207c5f. Latest consensus state: current_term: 1 leader_uuid: "a61fc15a84064ba0b1aad915f3207c5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a61fc15a84064ba0b1aad915f3207c5f" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36107 } } }
I20260806 04:39:46.570073 4928 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:46.570102 4929 sys_catalog.cc:458] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:46.571475 4938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:46.573988 4938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:46.579721 4938 catalog_manager.cc:1347] Generated new cluster ID: 0373de7012604ab08b09af47407c47b2
I20260806 04:39:46.579788 4938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:46.610111 4938 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:46.610494 4938 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:46.616209 4938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f: Generated new TSK 0
I20260806 04:39:46.616927 4938 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:39:46.629041 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36107
--builtin_ntp_servers=127.31.166.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 04:39:46.751784 4948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:46.752058 4948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:46.752097 4948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:46.756260 4948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:46.756326 4948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:46.756418 4948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:39:46.761199 4948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:46.762313 4948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:46.763779 4948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:46.773712 4953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:46.773852 4948 server_base.cc:1034] running on GCE node
W20260806 04:39:46.773705 4954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:46.773700 4956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:46.774403 4948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:46.775133 4948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:46.776331 4948 hybrid_clock.cc:648] HybridClock initialized: now 1785991186776314 us; error 28 us; skew 500 ppm
I20260806 04:39:46.778174 4948 webserver.cc:460] Webserver started at http://127.31.166.129:36439/ using document root <none> and password file <none>
I20260806 04:39:46.778767 4948 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:46.778864 4948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:46.779151 4948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:46.781317 4948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "8497d6133f944f76a3b8200b89a8bfa1"
format_stamp: "Formatted at 2026-08-06 04:39:46 on dist-test-slave-vkjp"
server_key: "c70fa59751214de5acc237aed81344b2"
server_key_iv: "ff3d07dee648fa89a8e271ad21829fe9"
server_key_version: "encryptionkey@0"
I20260806 04:39:46.781906 4948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "8497d6133f944f76a3b8200b89a8bfa1"
format_stamp: "Formatted at 2026-08-06 04:39:46 on dist-test-slave-vkjp"
server_key: "c70fa59751214de5acc237aed81344b2"
server_key_iv: "ff3d07dee648fa89a8e271ad21829fe9"
server_key_version: "encryptionkey@0"
I20260806 04:39:46.785595 4948 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260806 04:39:46.788223 4962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:46.789667 4948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:39:46.789837 4948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "8497d6133f944f76a3b8200b89a8bfa1"
format_stamp: "Formatted at 2026-08-06 04:39:46 on dist-test-slave-vkjp"
server_key: "c70fa59751214de5acc237aed81344b2"
server_key_iv: "ff3d07dee648fa89a8e271ad21829fe9"
server_key_version: "encryptionkey@0"
I20260806 04:39:46.789978 4948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:46.795127 4948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:46.795811 4948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:46.796025 4948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:46.796613 4948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:46.797590 4948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:46.797681 4948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:46.797758 4948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:46.797816 4948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:46.807286 4948 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:33365
I20260806 04:39:46.807329 5084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:33365 every 8 connection(s)
I20260806 04:39:46.808166 4948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:39:46.814395 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.809919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:37077 (local address 127.31.166.190:36107)
0806 04:39:46.810060 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:46.810064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:46.811139 (+ 1075us) server_negotiation.cc:407] Connection header received
0806 04:39:46.812609 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:46.812613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:46.812659 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:46.812744 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:46.813886 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:46.813890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:46.813892 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:46.813905 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:46.813928 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:46.813945 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:46.814044 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:46.814194 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 04:39:46.814243 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:39:46.815352 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.810276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36107 (local address 127.31.166.129:37077)
0806 04:39:46.810969 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:46.811011 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:46.812385 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:46.812914 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:46.812921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:46.813005 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:46.813616 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:39:46.813618 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:46.813657 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:39:46.813725 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:46.814035 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:46.814043 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:46.814050 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:39:46.814228 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 04:39:46.814464 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":127,"threads_started":1}
I20260806 04:39:46.815973 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 4948
I20260806 04:39:46.816107 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:39:46.816431 32410 external_mini_cluster.cc:1428] Setting key ed258fbd7b0b67cf86e81d84f2396e98
I20260806 04:39:46.816684 5085 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36107
I20260806 04:39:46.816928 5085 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:46.817346 5085 heartbeater.cc:511] Master 127.31.166.190:36107 requested a full tablet report, sending...
I20260806 04:39:46.818498 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36107
--builtin_ntp_servers=127.31.166.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 04:39:46.818847 4882 ts_manager.cc:194] Registered new tserver with Master: 8497d6133f944f76a3b8200b89a8bfa1 (127.31.166.129:33365)
W20260806 04:39:46.934331 5088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:46.934602 5088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:46.934643 5088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:46.938706 5088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:46.938767 5088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:46.938824 5088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:39:46.943598 5088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:46.944667 5088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:46.946026 5088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:46.956122 5098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:46.956120 5093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:46.956198 5088 server_base.cc:1034] running on GCE node
W20260806 04:39:46.956090 5096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:46.956868 5088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:46.957538 5088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:46.958760 5088 hybrid_clock.cc:648] HybridClock initialized: now 1785991186958747 us; error 28 us; skew 500 ppm
I20260806 04:39:46.960812 5088 webserver.cc:460] Webserver started at http://127.31.166.130:44031/ using document root <none> and password file <none>
I20260806 04:39:46.961414 5088 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:46.961516 5088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:46.961762 5088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:46.964007 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "536756ca3d7f4c2db155c1dcca909ca6"
format_stamp: "Formatted at 2026-08-06 04:39:46 on dist-test-slave-vkjp"
server_key: "40aa427cc44f6ee9062e42e2e750e556"
server_key_iv: "8d8dd23c9c098aaaa4cc4db9e3af4732"
server_key_version: "encryptionkey@0"
I20260806 04:39:46.964787 5088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "536756ca3d7f4c2db155c1dcca909ca6"
format_stamp: "Formatted at 2026-08-06 04:39:46 on dist-test-slave-vkjp"
server_key: "40aa427cc44f6ee9062e42e2e750e556"
server_key_iv: "8d8dd23c9c098aaaa4cc4db9e3af4732"
server_key_version: "encryptionkey@0"
I20260806 04:39:46.968407 5088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:39:46.971246 5105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:46.972469 5088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:46.972563 5088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "536756ca3d7f4c2db155c1dcca909ca6"
format_stamp: "Formatted at 2026-08-06 04:39:46 on dist-test-slave-vkjp"
server_key: "40aa427cc44f6ee9062e42e2e750e556"
server_key_iv: "8d8dd23c9c098aaaa4cc4db9e3af4732"
server_key_version: "encryptionkey@0"
I20260806 04:39:46.972652 5088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:46.977010 5088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:46.977555 5088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:46.977685 5088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:46.978156 5088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:46.979031 5088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:46.979127 5088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:46.979195 5088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:46.979216 5088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:46.987836 5088 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:37197
I20260806 04:39:46.987911 5225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:37197 every 8 connection(s)
I20260806 04:39:46.988806 5088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:39:46.995033 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 5088
I20260806 04:39:46.995225 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:39:46.995368 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.990590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:56201 (local address 127.31.166.190:36107)
0806 04:39:46.990738 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:46.990742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:46.991803 (+ 1061us) server_negotiation.cc:407] Connection header received
0806 04:39:46.993080 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:46.993083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:46.993123 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:46.993200 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:46.994622 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:46.994625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:46.994627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:46.994641 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:46.994682 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:46.994693 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:46.994848 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:46.995024 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 04:39:46.995133 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:39:46.995600 32410 external_mini_cluster.cc:1428] Setting key 6a806856ee6544c32c0468c8cd7acf7c
I20260806 04:39:46.996356 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:46.990907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36107 (local address 127.31.166.130:56201)
0806 04:39:46.991638 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:46.991680 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:46.992887 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:46.993566 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:46.993574 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:46.993660 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:46.994189 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:39:46.994191 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:46.994232 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:39:46.994413 (+ 181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:46.994837 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:46.994847 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:46.994854 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:39:46.995070 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 04:39:46.995383 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":173,"threads_started":1}
I20260806 04:39:46.997709 5226 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36107
I20260806 04:39:46.997984 5226 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:46.998044 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36107
--builtin_ntp_servers=127.31.166.148:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 04:39:46.998466 5226 heartbeater.cc:511] Master 127.31.166.190:36107 requested a full tablet report, sending...
I20260806 04:39:46.999740 4876 ts_manager.cc:194] Registered new tserver with Master: 536756ca3d7f4c2db155c1dcca909ca6 (127.31.166.130:37197)
W20260806 04:39:47.113391 5229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:47.113701 5229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:47.113782 5229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:47.117821 5229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:47.117908 5229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:47.118000 5229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:39:47.122745 5229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:47.123979 5229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:47.125478 5229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:47.135140 5235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:47.135160 5236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:47.135425 5229 server_base.cc:1034] running on GCE node
W20260806 04:39:47.135152 5238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:47.135885 5229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:47.136593 5229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:47.137781 5229 hybrid_clock.cc:648] HybridClock initialized: now 1785991187137747 us; error 43 us; skew 500 ppm
I20260806 04:39:47.139835 5229 webserver.cc:460] Webserver started at http://127.31.166.131:46513/ using document root <none> and password file <none>
I20260806 04:39:47.140476 5229 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:47.140585 5229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:47.140861 5229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:47.143299 5229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "f4f8ce91cce94c2a94a1f11292196873"
format_stamp: "Formatted at 2026-08-06 04:39:47 on dist-test-slave-vkjp"
server_key: "f6bb5145d44b764f1c266c6453fb392e"
server_key_iv: "6984f5311d2562d119d476901d708af9"
server_key_version: "encryptionkey@0"
I20260806 04:39:47.143961 5229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "f4f8ce91cce94c2a94a1f11292196873"
format_stamp: "Formatted at 2026-08-06 04:39:47 on dist-test-slave-vkjp"
server_key: "f6bb5145d44b764f1c266c6453fb392e"
server_key_iv: "6984f5311d2562d119d476901d708af9"
server_key_version: "encryptionkey@0"
I20260806 04:39:47.148241 5229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:39:47.150982 5244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:47.152671 5229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:39:47.152827 5229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "f4f8ce91cce94c2a94a1f11292196873"
format_stamp: "Formatted at 2026-08-06 04:39:47 on dist-test-slave-vkjp"
server_key: "f6bb5145d44b764f1c266c6453fb392e"
server_key_iv: "6984f5311d2562d119d476901d708af9"
server_key_version: "encryptionkey@0"
I20260806 04:39:47.152956 5229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:47.158056 5229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:47.158695 5229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:47.158895 5229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:47.159570 5229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:47.160533 5229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:47.160610 5229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:47.160692 5229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:47.160750 5229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:47.170686 5229 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:38659
I20260806 04:39:47.170724 5365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:38659 every 8 connection(s)
I20260806 04:39:47.171720 5229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:39:47.175511 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 5229
I20260806 04:39:47.175663 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:39:47.176021 32410 external_mini_cluster.cc:1428] Setting key dc917b6ffe615c65360c464e79d11304
I20260806 04:39:47.178815 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.173663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:41259 (local address 127.31.166.190:36107)
0806 04:39:47.173823 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:47.173827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:47.174909 (+ 1082us) server_negotiation.cc:407] Connection header received
0806 04:39:47.176635 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:47.176639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:47.176690 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:47.176782 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:47.178206 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:47.178229 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:47.178236 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:47.178251 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:47.178279 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:47.178351 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:47.178449 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:47.178654 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 04:39:47.178705 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:39:47.179785 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.174059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36107 (local address 127.31.166.131:41259)
0806 04:39:47.174718 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:47.174759 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:47.176387 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:47.177011 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:47.177021 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:47.177138 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:47.177889 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:39:47.177892 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:47.177951 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:39:47.178040 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:47.178462 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:47.178472 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:47.178478 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 04:39:47.178681 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 04:39:47.178958 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":107,"threads_started":1}
I20260806 04:39:47.180966 5366 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36107
I20260806 04:39:47.181221 5366 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:47.181643 5366 heartbeater.cc:511] Master 127.31.166.190:36107 requested a full tablet report, sending...
I20260806 04:39:47.182642 4876 ts_manager.cc:194] Registered new tserver with Master: f4f8ce91cce94c2a94a1f11292196873 (127.31.166.131:38659)
I20260806 04:39:47.190255 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:39:47.193652 4925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.192213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59808 (local address 127.31.166.190:36107)
0806 04:39:47.192336 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:47.192339 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:47.192513 (+ 174us) server_negotiation.cc:407] Connection header received
0806 04:39:47.192609 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:47.192611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:47.192646 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:47.192707 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:47.193390 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:47.193392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:47.193394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:47.193402 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:47.193417 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:47.193424 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:47.193487 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:47.193528 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 04:39:47.193557 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:39:47.196696 4876 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:39:47.199755 4876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:39:47.211740 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.209050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.31.166.129:33365)
0806 04:39:47.209967 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:47.209975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:47.210018 (+ 43us) server_negotiation.cc:407] Connection header received
0806 04:39:47.210074 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:47.210079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:47.210242 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:47.210397 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:47.211158 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:47.211165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:47.211177 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:47.211191 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:47.211243 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:47.211284 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:47.211417 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:47.211524 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 04:39:47.211595 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":161,"threads_started":1}
I20260806 04:39:47.211725 5381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.208498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:33365 (local address 127.0.0.1:33612)
0806 04:39:47.209639 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:47.209651 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:47.209727 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:47.210445 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:47.210447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:47.210462 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:47.210922 (+ 460us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:39:47.210926 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:47.210942 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:39:47.210999 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:47.211391 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:47.211400 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:47.211406 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 04:39:47.211511 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 04:39:47.211580 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"spinlock_wait_cycles":1408,"thread_start_us":116,"threads_started":1}
I20260806 04:39:47.212193 5382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.208314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:38659 (local address 127.0.0.1:37310)
0806 04:39:47.209266 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:47.209315 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:47.209427 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:47.210087 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:47.210093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:47.210169 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:47.210909 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:39:47.210911 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:47.210930 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:39:47.210998 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:47.211594 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:47.211597 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:47.211600 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 04:39:47.212032 (+ 432us) client_negotiation.cc:240] Negotiation successful
0806 04:39:47.212064 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":73,"threads_started":1}
I20260806 04:39:47.212726 5379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.208733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:37197 (local address 127.0.0.1:58440)
0806 04:39:47.209637 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:47.209651 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:47.209727 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:47.210675 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:47.210681 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:47.210700 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:47.211165 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:39:47.211167 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:47.211176 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:39:47.211197 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:47.211594 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:47.211597 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:47.211600 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 04:39:47.211676 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 04:39:47.211713 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"spinlock_wait_cycles":2304,"thread_start_us":108,"threads_started":1}
I20260806 04:39:47.212725 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.208492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.31.166.131:38659)
0806 04:39:47.209541 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:47.209555 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:47.209580 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:39:47.209693 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:47.209719 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:47.209913 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:47.210066 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:47.211406 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:47.211410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:47.211418 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:47.211428 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:47.211462 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:47.211486 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:47.212388 (+ 902us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:47.212435 (+ 47us) server_negotiation.cc:299] Negotiation successful
0806 04:39:47.212495 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":106,"threads_started":1}
I20260806 04:39:47.213836 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.209081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.31.166.130:37197)
0806 04:39:47.209519 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:47.209533 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:47.209752 (+ 219us) server_negotiation.cc:407] Connection header received
0806 04:39:47.209817 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:47.209822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:47.209958 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:47.210075 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:47.211328 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:47.211334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:47.211345 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:47.211359 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:47.211401 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:47.211434 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:47.211549 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:47.213609 (+ 2060us) server_negotiation.cc:299] Negotiation successful
0806 04:39:47.213692 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":98,"threads_started":1}
I20260806 04:39:47.215262 5295 tablet_service.cc:1467] Processing CreateTablet for tablet d93d8f3482cb4ac396f2e60ee7ae50ef (DEFAULT_TABLE table=test-table [id=7b36bc7db0554f829c196be053b8dfe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:47.215260 5015 tablet_service.cc:1467] Processing CreateTablet for tablet d93d8f3482cb4ac396f2e60ee7ae50ef (DEFAULT_TABLE table=test-table [id=7b36bc7db0554f829c196be053b8dfe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:47.216338 5015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93d8f3482cb4ac396f2e60ee7ae50ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:47.216365 5295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93d8f3482cb4ac396f2e60ee7ae50ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:47.216323 5155 tablet_service.cc:1467] Processing CreateTablet for tablet d93d8f3482cb4ac396f2e60ee7ae50ef (DEFAULT_TABLE table=test-table [id=7b36bc7db0554f829c196be053b8dfe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:47.217265 5155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93d8f3482cb4ac396f2e60ee7ae50ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:47.222572 5386 tablet_bootstrap.cc:492] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1: Bootstrap starting.
I20260806 04:39:47.224354 5388 tablet_bootstrap.cc:492] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6: Bootstrap starting.
I20260806 04:39:47.224782 5387 tablet_bootstrap.cc:492] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873: Bootstrap starting.
I20260806 04:39:47.225122 5386 tablet_bootstrap.cc:654] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:47.225865 5386 log.cc:824] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:47.226517 5388 tablet_bootstrap.cc:654] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:47.226732 5387 tablet_bootstrap.cc:654] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:47.227404 5388 log.cc:824] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:47.227502 5387 log.cc:824] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:47.228026 5386 tablet_bootstrap.cc:492] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1: No bootstrap required, opened a new log
I20260806 04:39:47.228241 5386 ts_tablet_manager.cc:1397] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:39:47.229154 5388 tablet_bootstrap.cc:492] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6: No bootstrap required, opened a new log
I20260806 04:39:47.229346 5388 ts_tablet_manager.cc:1397] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:39:47.232071 5386 raft_consensus.cc:348] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f8ce91cce94c2a94a1f11292196873" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38659 } } peers { permanent_uuid: "8497d6133f944f76a3b8200b89a8bfa1" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 33365 } } peers { permanent_uuid: "536756ca3d7f4c2db155c1dcca909ca6" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37197 } }
I20260806 04:39:47.232288 5386 raft_consensus.cc:374] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:47.232365 5386 raft_consensus.cc:729] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8497d6133f944f76a3b8200b89a8bfa1, State: Initialized, Role: FOLLOWER
I20260806 04:39:47.232764 5386 consensus_queue.cc:260] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f8ce91cce94c2a94a1f11292196873" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38659 } } peers { permanent_uuid: "8497d6133f944f76a3b8200b89a8bfa1" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 33365 } } peers { permanent_uuid: "536756ca3d7f4c2db155c1dcca909ca6" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37197 } }
I20260806 04:39:47.233072 5388 raft_consensus.cc:348] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f8ce91cce94c2a94a1f11292196873" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38659 } } peers { permanent_uuid: "8497d6133f944f76a3b8200b89a8bfa1" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 33365 } } peers { permanent_uuid: "536756ca3d7f4c2db155c1dcca909ca6" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37197 } }
I20260806 04:39:47.233281 5388 raft_consensus.cc:374] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:47.233338 5388 raft_consensus.cc:729] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 536756ca3d7f4c2db155c1dcca909ca6, State: Initialized, Role: FOLLOWER
I20260806 04:39:47.233402 5085 heartbeater.cc:503] Master 127.31.166.190:36107 was elected leader, sending a full tablet report...
I20260806 04:39:47.233777 5386 ts_tablet_manager.cc:1428] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 04:39:47.233777 5388 consensus_queue.cc:260] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f8ce91cce94c2a94a1f11292196873" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38659 } } peers { permanent_uuid: "8497d6133f944f76a3b8200b89a8bfa1" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 33365 } } peers { permanent_uuid: "536756ca3d7f4c2db155c1dcca909ca6" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37197 } }
I20260806 04:39:47.234395 5226 heartbeater.cc:503] Master 127.31.166.190:36107 was elected leader, sending a full tablet report...
I20260806 04:39:47.234747 5388 ts_tablet_manager.cc:1428] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 04:39:47.236143 5387 tablet_bootstrap.cc:492] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873: No bootstrap required, opened a new log
I20260806 04:39:47.236400 5387 ts_tablet_manager.cc:1397] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
W20260806 04:39:47.239769 5227 tablet.cc:2369] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:39:47.240933 5387 raft_consensus.cc:348] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f8ce91cce94c2a94a1f11292196873" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38659 } } peers { permanent_uuid: "8497d6133f944f76a3b8200b89a8bfa1" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 33365 } } peers { permanent_uuid: "536756ca3d7f4c2db155c1dcca909ca6" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37197 } }
I20260806 04:39:47.241180 5387 raft_consensus.cc:374] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:47.241216 5387 raft_consensus.cc:729] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4f8ce91cce94c2a94a1f11292196873, State: Initialized, Role: FOLLOWER
I20260806 04:39:47.241571 5387 consensus_queue.cc:260] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f8ce91cce94c2a94a1f11292196873" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38659 } } peers { permanent_uuid: "8497d6133f944f76a3b8200b89a8bfa1" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 33365 } } peers { permanent_uuid: "536756ca3d7f4c2db155c1dcca909ca6" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37197 } }
I20260806 04:39:47.242175 5366 heartbeater.cc:503] Master 127.31.166.190:36107 was elected leader, sending a full tablet report...
I20260806 04:39:47.242408 5387 ts_tablet_manager.cc:1428] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260806 04:39:47.309309 5086 tablet.cc:2369] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:39:47.422832 5367 tablet.cc:2369] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:39:47.490752 5395 raft_consensus.cc:482] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:39:47.491011 5395 raft_consensus.cc:504] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f8ce91cce94c2a94a1f11292196873" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38659 } } peers { permanent_uuid: "8497d6133f944f76a3b8200b89a8bfa1" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 33365 } } peers { permanent_uuid: "536756ca3d7f4c2db155c1dcca909ca6" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37197 } }
I20260806 04:39:47.492399 5395 leader_election.cc:288] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8497d6133f944f76a3b8200b89a8bfa1 (127.31.166.129:33365), 536756ca3d7f4c2db155c1dcca909ca6 (127.31.166.130:37197)
I20260806 04:39:47.494470 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.492660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:37197 (local address 127.31.166.131:53237)
0806 04:39:47.492866 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:47.492882 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:47.492990 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:47.493275 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:47.493278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:47.493293 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:47.493857 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:39:47.493861 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:47.493876 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:39:47.493909 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:47.494194 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:47.494200 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:47.494205 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:39:47.494282 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 04:39:47.494319 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260806 04:39:47.494539 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.492710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:53237 (local address 127.31.166.130:37197)
0806 04:39:47.492875 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:47.492880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:47.492972 (+ 92us) server_negotiation.cc:407] Connection header received
0806 04:39:47.493110 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:47.493116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:47.493162 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:47.493245 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:47.494062 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:47.494065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:47.494069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:47.494082 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:47.494103 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:47.494113 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:47.494203 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:47.494280 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 04:39:47.494325 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:39:47.494659 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.492788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:56863 (local address 127.31.166.129:33365)
0806 04:39:47.492947 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:47.492952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:47.493165 (+ 213us) server_negotiation.cc:407] Connection header received
0806 04:39:47.493268 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:47.493271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:47.493312 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:47.493401 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:47.494261 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:47.494266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:47.494269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:47.494282 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:47.494307 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:47.494316 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:47.494398 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:47.494487 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 04:39:47.494528 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 04:39:47.494797 5396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.492675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:33365 (local address 127.31.166.131:56863)
0806 04:39:47.493035 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:47.493069 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:47.493161 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:47.493424 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:47.493426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:47.493438 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:47.494061 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:39:47.494063 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:47.494073 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:39:47.494098 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:47.494402 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:47.494405 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:47.494408 (+ 3us) client_negotiation.cc:769] Sending connection context
0806 04:39:47.494489 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 04:39:47.494659 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":15,"thread_start_us":117,"threads_started":1}
I20260806 04:39:47.495182 5177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93d8f3482cb4ac396f2e60ee7ae50ef" candidate_uuid: "f4f8ce91cce94c2a94a1f11292196873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "536756ca3d7f4c2db155c1dcca909ca6" is_pre_election: true
I20260806 04:39:47.495488 5177 raft_consensus.cc:2454] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f8ce91cce94c2a94a1f11292196873 in term 0.
I20260806 04:39:47.495668 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93d8f3482cb4ac396f2e60ee7ae50ef" candidate_uuid: "f4f8ce91cce94c2a94a1f11292196873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8497d6133f944f76a3b8200b89a8bfa1" is_pre_election: true
I20260806 04:39:47.495954 5037 raft_consensus.cc:2454] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4f8ce91cce94c2a94a1f11292196873 in term 0.
I20260806 04:39:47.496042 5245 leader_election.cc:302] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 536756ca3d7f4c2db155c1dcca909ca6, f4f8ce91cce94c2a94a1f11292196873; no voters:
I20260806 04:39:47.496364 5395 raft_consensus.cc:2781] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:39:47.496446 5395 raft_consensus.cc:482] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:39:47.496500 5395 raft_consensus.cc:3037] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:47.497627 5395 raft_consensus.cc:504] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f8ce91cce94c2a94a1f11292196873" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38659 } } peers { permanent_uuid: "8497d6133f944f76a3b8200b89a8bfa1" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 33365 } } peers { permanent_uuid: "536756ca3d7f4c2db155c1dcca909ca6" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37197 } }
I20260806 04:39:47.497987 5395 leader_election.cc:288] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [CANDIDATE]: Term 1 election: Requested vote from peers 8497d6133f944f76a3b8200b89a8bfa1 (127.31.166.129:33365), 536756ca3d7f4c2db155c1dcca909ca6 (127.31.166.130:37197)
I20260806 04:39:47.498292 5177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93d8f3482cb4ac396f2e60ee7ae50ef" candidate_uuid: "f4f8ce91cce94c2a94a1f11292196873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "536756ca3d7f4c2db155c1dcca909ca6"
I20260806 04:39:47.498426 5177 raft_consensus.cc:3037] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:47.499282 5037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93d8f3482cb4ac396f2e60ee7ae50ef" candidate_uuid: "f4f8ce91cce94c2a94a1f11292196873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8497d6133f944f76a3b8200b89a8bfa1"
I20260806 04:39:47.499436 5037 raft_consensus.cc:3037] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:47.499606 5177 raft_consensus.cc:2454] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f8ce91cce94c2a94a1f11292196873 in term 1.
I20260806 04:39:47.499919 5245 leader_election.cc:302] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 536756ca3d7f4c2db155c1dcca909ca6, f4f8ce91cce94c2a94a1f11292196873; no voters:
I20260806 04:39:47.500082 5395 raft_consensus.cc:2781] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:47.500298 5395 raft_consensus.cc:686] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [term 1 LEADER]: Becoming Leader. State: Replica: f4f8ce91cce94c2a94a1f11292196873, State: Running, Role: LEADER
I20260806 04:39:47.500639 5395 consensus_queue.cc:237] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f8ce91cce94c2a94a1f11292196873" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38659 } } peers { permanent_uuid: "8497d6133f944f76a3b8200b89a8bfa1" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 33365 } } peers { permanent_uuid: "536756ca3d7f4c2db155c1dcca909ca6" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37197 } }
I20260806 04:39:47.500765 5037 raft_consensus.cc:2454] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4f8ce91cce94c2a94a1f11292196873 in term 1.
I20260806 04:39:47.503643 4876 catalog_manager.cc:5495] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4f8ce91cce94c2a94a1f11292196873 (127.31.166.131). New cstate: current_term: 1 leader_uuid: "f4f8ce91cce94c2a94a1f11292196873" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4f8ce91cce94c2a94a1f11292196873" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8497d6133f944f76a3b8200b89a8bfa1" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 33365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "536756ca3d7f4c2db155c1dcca909ca6" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37197 } health_report { overall_health: UNKNOWN } } }
I20260806 04:39:47.575485 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.573750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.31.166.131:38659)
0806 04:39:47.573918 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:47.573922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:47.573940 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:39:47.574054 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:47.574057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:47.574125 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:47.574212 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:47.574933 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:47.574936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:47.574938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:47.574951 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:47.574974 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:47.574984 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:47.575147 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:47.575204 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 04:39:47.575255 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 04:39:47.581691 5037 raft_consensus.cc:1261] T d93d8f3482cb4ac396f2e60ee7ae50ef P 8497d6133f944f76a3b8200b89a8bfa1 [term 1 FOLLOWER]: Refusing update from remote peer f4f8ce91cce94c2a94a1f11292196873: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:47.581691 5177 raft_consensus.cc:1261] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6 [term 1 FOLLOWER]: Refusing update from remote peer f4f8ce91cce94c2a94a1f11292196873: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:47.582455 5397 consensus_queue.cc:1035] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8497d6133f944f76a3b8200b89a8bfa1" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 33365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:47.582631 5395 consensus_queue.cc:1035] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [LEADER]: Connected to new peer: Peer: permanent_uuid: "536756ca3d7f4c2db155c1dcca909ca6" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:47.590112 5401 mvcc.cc:205] Tried to move back new op lower bound from 7315419904327393280 to 7315419904004976640. Current Snapshot: MvccSnapshot[applied={T|T < 7315419904327393280}]
I20260806 04:39:47.598115 5402 mvcc.cc:205] Tried to move back new op lower bound from 7315419904327393280 to 7315419904004976640. Current Snapshot: MvccSnapshot[applied={T|T < 7315419904327393280}]
I20260806 04:39:47.599390 4882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:47.599574 4882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:47.602078 4882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a61fc15a84064ba0b1aad915f3207c5f: Sending DeleteTablet for 3 replicas of tablet d93d8f3482cb4ac396f2e60ee7ae50ef
I20260806 04:39:47.602808 5295 tablet_service.cc:1514] Processing DeleteTablet for tablet d93d8f3482cb4ac396f2e60ee7ae50ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:47 UTC) from {username='slave'} at 127.0.0.1:37310
I20260806 04:39:47.602846 5015 tablet_service.cc:1514] Processing DeleteTablet for tablet d93d8f3482cb4ac396f2e60ee7ae50ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:47 UTC) from {username='slave'} at 127.0.0.1:33612
I20260806 04:39:47.603240 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet d93d8f3482cb4ac396f2e60ee7ae50ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:47 UTC) from {username='slave'} at 127.0.0.1:58440
I20260806 04:39:47.603737 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 4948
I20260806 04:39:47.608506 5412 tablet_replica.cc:330] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6: stopping tablet replica
I20260806 04:39:47.608839 5412 raft_consensus.cc:2229] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:47.609181 5412 raft_consensus.cc:2258] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:47.610644 5412 ts_tablet_manager.cc:1905] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:47.612982 5412 ts_tablet_manager.cc:1918] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:39:47.613101 5412 log.cc:1196] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/d93d8f3482cb4ac396f2e60ee7ae50ef
I20260806 04:39:47.613435 5412 ts_tablet_manager.cc:1939] T d93d8f3482cb4ac396f2e60ee7ae50ef P 536756ca3d7f4c2db155c1dcca909ca6: Deleting consensus metadata
I20260806 04:39:47.613466 5410 tablet_replica.cc:330] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873: stopping tablet replica
I20260806 04:39:47.614037 5410 raft_consensus.cc:2229] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:39:47.614468 5410 raft_consensus.cc:2258] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:47.616285 5410 ts_tablet_manager.cc:1905] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:47.616389 4867 catalog_manager.cc:4841] TS 536756ca3d7f4c2db155c1dcca909ca6 (127.31.166.130:37197): tablet d93d8f3482cb4ac396f2e60ee7ae50ef (table test-table [id=7b36bc7db0554f829c196be053b8dfe3]) successfully deleted
I20260806 04:39:47.619980 5410 ts_tablet_manager.cc:1918] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:39:47.620091 5410 log.cc:1196] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/d93d8f3482cb4ac396f2e60ee7ae50ef
I20260806 04:39:47.620448 5410 ts_tablet_manager.cc:1939] T d93d8f3482cb4ac396f2e60ee7ae50ef P f4f8ce91cce94c2a94a1f11292196873: Deleting consensus metadata
W20260806 04:39:47.621822 4870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:33365 (error 108)
I20260806 04:39:47.621851 4868 catalog_manager.cc:4841] TS f4f8ce91cce94c2a94a1f11292196873 (127.31.166.131:38659): tablet d93d8f3482cb4ac396f2e60ee7ae50ef (table test-table [id=7b36bc7db0554f829c196be053b8dfe3]) successfully deleted
I20260806 04:39:47.622802 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 5088
I20260806 04:39:47.623039 5379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.622696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:33365 (local address 127.0.0.1:33620)
0806 04:39:47.622833 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:47.622932 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 04:39:47.629640 4870 catalog_manager.cc:4584] TS 8497d6133f944f76a3b8200b89a8bfa1 (127.31.166.129:33365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d93d8f3482cb4ac396f2e60ee7ae50ef: Network error: Client connection negotiation failed: client connection to 127.31.166.129:33365: connect: Connection refused (error 111)
I20260806 04:39:47.635736 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 5229
I20260806 04:39:47.642537 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 4851
2026-08-06T04:39:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1287 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 04:39:47.664970 32410 mini_oidc.cc:152] Starting JWKS server
I20260806 04:39:47.666587 32410 webserver.cc:460] Webserver started at https://127.0.0.1:45533/ using document root <none> and password file <none>
I20260806 04:39:47.666728 32410 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 04:39:47.666957 32410 webserver.cc:460] Webserver started at http://127.0.0.1:33601/ using document root <none> and password file <none>
2026-08-06T04:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:47Z Disabled control of system clock
I20260806 04:39:47.676326 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:45093
--webserver_interface=127.31.166.190
--webserver_port=0
--builtin_ntp_servers=127.31.166.148:42417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:45093
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45533/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/testchainca.cert with env {}
W20260806 04:39:47.787788 5420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:47.788057 5420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:47.788097 5420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:47.792022 5420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:47.792073 5420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:47.792088 5420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:47.792105 5420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:47.792120 5420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 04:39:47.792135 5420 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 04:39:47.792150 5420 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45533/jwks.json
I20260806 04:39:47.797394 5420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:42417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:45093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:45093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45533/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:47.798399 5420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:47.799364 5420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:47.807641 5425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:47.807849 5420 server_base.cc:1034] running on GCE node
W20260806 04:39:47.807667 5426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:47.807710 5428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:47.808804 5420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:47.809937 5420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:47.811153 5420 hybrid_clock.cc:648] HybridClock initialized: now 1785991187811106 us; error 59 us; skew 500 ppm
I20260806 04:39:47.813177 5420 webserver.cc:460] Webserver started at http://127.31.166.190:40963/ using document root <none> and password file <none>
I20260806 04:39:47.813795 5420 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:47.813863 5420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:47.814095 5420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:47.816293 5420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "d0e57f8aa82942839a6dc2e0b50933d1"
format_stamp: "Formatted at 2026-08-06 04:39:47 on dist-test-slave-vkjp"
server_key: "a8d221660be5d3540c31f71c16d25418"
server_key_iv: "335846923a497c5a9b95b0aa966110b3"
server_key_version: "encryptionkey@0"
I20260806 04:39:47.816932 5420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "d0e57f8aa82942839a6dc2e0b50933d1"
format_stamp: "Formatted at 2026-08-06 04:39:47 on dist-test-slave-vkjp"
server_key: "a8d221660be5d3540c31f71c16d25418"
server_key_iv: "335846923a497c5a9b95b0aa966110b3"
server_key_version: "encryptionkey@0"
I20260806 04:39:47.820888 5420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:39:47.823988 5434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:47.825237 5420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:39:47.825414 5420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "d0e57f8aa82942839a6dc2e0b50933d1"
format_stamp: "Formatted at 2026-08-06 04:39:47 on dist-test-slave-vkjp"
server_key: "a8d221660be5d3540c31f71c16d25418"
server_key_iv: "335846923a497c5a9b95b0aa966110b3"
server_key_version: "encryptionkey@0"
I20260806 04:39:47.825546 5420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:47.837426 5420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:47.838167 5420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:47.838379 5420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:47.845997 5420 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:45093
I20260806 04:39:47.846026 5491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:45093 every 8 connection(s)
I20260806 04:39:47.846985 5420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:47.850459 5492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:47.851989 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 5420
I20260806 04:39:47.852154 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:47.852475 32410 external_mini_cluster.cc:1428] Setting key 82f80b4c21cff97e261bdd363cf87e32
I20260806 04:39:47.857057 5492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1: Bootstrap starting.
I20260806 04:39:47.859483 5492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:47.860400 5492 log.cc:824] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:47.862161 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.853968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.31.166.190:45093)
0806 04:39:47.854658 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:47.854673 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:47.854711 (+ 38us) server_negotiation.cc:407] Connection header received
0806 04:39:47.855465 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:47.855483 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:47.855895 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:47.856232 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:47.856675 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:47.857718 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:47.858640 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:47.859197 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:47.859366 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:47.860305 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:47.860311 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:47.860324 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:47.860790 (+ 466us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:47.860864 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:47.860903 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:47.861023 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:47.861190 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 04:39:47.861443 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":149,"threads_started":1}
I20260806 04:39:47.862637 5492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1: No bootstrap required, opened a new log
I20260806 04:39:47.865234 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:39:47.866699 5492 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e57f8aa82942839a6dc2e0b50933d1" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45093 } }
I20260806 04:39:47.867033 5492 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:47.867170 5492 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e57f8aa82942839a6dc2e0b50933d1, State: Initialized, Role: FOLLOWER
I20260806 04:39:47.867609 5492 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e57f8aa82942839a6dc2e0b50933d1" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45093 } }
I20260806 04:39:47.867772 5492 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:47.867820 5492 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:47.867895 5492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:47.868871 5492 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e57f8aa82942839a6dc2e0b50933d1" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45093 } }
I20260806 04:39:47.869189 5492 leader_election.cc:302] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0e57f8aa82942839a6dc2e0b50933d1; no voters:
I20260806 04:39:47.869534 5492 leader_election.cc:288] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:47.869679 5497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:47.869969 5497 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [term 1 LEADER]: Becoming Leader. State: Replica: d0e57f8aa82942839a6dc2e0b50933d1, State: Running, Role: LEADER
I20260806 04:39:47.870285 5497 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e57f8aa82942839a6dc2e0b50933d1" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45093 } }
I20260806 04:39:47.870919 5492 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:47.872223 5499 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0e57f8aa82942839a6dc2e0b50933d1. Latest consensus state: current_term: 1 leader_uuid: "d0e57f8aa82942839a6dc2e0b50933d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e57f8aa82942839a6dc2e0b50933d1" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45093 } } }
I20260806 04:39:47.872244 5498 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0e57f8aa82942839a6dc2e0b50933d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0e57f8aa82942839a6dc2e0b50933d1" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45093 } } }
I20260806 04:39:47.872490 5499 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:47.872519 5498 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:47.872921 5506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:47.876361 5506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:47.882293 5506 catalog_manager.cc:1347] Generated new cluster ID: e1551f7a6ffb40e19fbd3eb231b9f026
I20260806 04:39:47.882387 5506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:47.911832 5506 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:47.913187 5506 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:47.922560 5506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0e57f8aa82942839a6dc2e0b50933d1: Generated new TSK 0
I20260806 04:39:47.923485 5506 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 04:39:47.957508 5522 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 04:39:47.957712 5522 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 04:39:47.958359 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:47.940053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52058 (local address 127.31.166.190:45093)
0806 04:39:47.940192 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:47.940197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:47.940368 (+ 171us) server_negotiation.cc:407] Connection header received
0806 04:39:47.940574 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:47.940577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:47.940670 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:47.940823 (+ 153us) server_negotiation.cc:226] Negotiated authn=JWT
0806 04:39:47.941093 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:47.941933 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:47.942816 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:47.943199 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:47.943307 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:47.943667 (+ 360us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 04:39:47.957907 (+ 14240us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45533/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 04:39:47.958209 (+ 302us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52058: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45533/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 04:39:47.958429 5495 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52058: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45533/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 04:39:47.959781 5521 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:39:47.939923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:45093 (local address 127.0.0.1:52058)
0806 04:39:47.940277 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:47.940291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:47.940408 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:47.940840 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:47.940842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:47.940850 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 04:39:47.940971 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:47.940976 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:47.942064 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:47.942066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:47.942699 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:47.942707 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:47.943322 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:47.943325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:47.943402 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:47.943451 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 04:39:47.958182 (+ 14731us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45533/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 04:39:47.958405 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.166.190:45093: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45533/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":128,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.166.190:45093: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45533/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 04:39:47.960597 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 5420
2026-08-06T04:39:47Z chronyd exiting
I20260806 04:39:47.999619 32410 test_util.cc:175] -----------------------------------------------
I20260806 04:39:47.999727 32410 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785991160137184-32410-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (336 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 04:39:48.001075 32410 mini_oidc.cc:152] Starting JWKS server
I20260806 04:39:48.002059 32410 webserver.cc:460] Webserver started at https://127.0.0.1:36343/ using document root <none> and password file <none>
I20260806 04:39:48.002180 32410 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 04:39:48.002485 32410 webserver.cc:460] Webserver started at http://127.0.0.1:41163/ using document root <none> and password file <none>
2026-08-06T04:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:48Z Disabled control of system clock
I20260806 04:39:48.012622 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:44027
--webserver_interface=127.31.166.190
--webserver_port=0
--builtin_ntp_servers=127.31.166.148:41625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:44027
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36343/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/testchainca.cert with env {}
W20260806 04:39:48.129460 5528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:48.129806 5528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:48.129904 5528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:48.134081 5528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:48.134194 5528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:48.134228 5528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:48.134267 5528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:48.134305 5528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 04:39:48.134346 5528 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 04:39:48.134382 5528 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36343/jwks.json
I20260806 04:39:48.140220 5528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:41625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:44027
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:44027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36343/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:48.141430 5528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:48.142519 5528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:48.151691 5536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:48.151748 5533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:48.151962 5528 server_base.cc:1034] running on GCE node
W20260806 04:39:48.151753 5534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:48.152581 5528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:48.153769 5528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:48.154974 5528 hybrid_clock.cc:648] HybridClock initialized: now 1785991188154944 us; error 45 us; skew 500 ppm
I20260806 04:39:48.156956 5528 webserver.cc:460] Webserver started at http://127.31.166.190:45645/ using document root <none> and password file <none>
I20260806 04:39:48.157557 5528 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:48.157655 5528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:48.157903 5528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:48.160087 5528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "29d0d5ec9da54d148c47f51025cbf3cc"
format_stamp: "Formatted at 2026-08-06 04:39:48 on dist-test-slave-vkjp"
server_key: "8afa069670efc6db6461b4db5c8f9ce9"
server_key_iv: "c2d73953ad8d2981bc4b12ef34358d08"
server_key_version: "encryptionkey@0"
I20260806 04:39:48.160673 5528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "29d0d5ec9da54d148c47f51025cbf3cc"
format_stamp: "Formatted at 2026-08-06 04:39:48 on dist-test-slave-vkjp"
server_key: "8afa069670efc6db6461b4db5c8f9ce9"
server_key_iv: "c2d73953ad8d2981bc4b12ef34358d08"
server_key_version: "encryptionkey@0"
I20260806 04:39:48.164587 5528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:39:48.167294 5543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:48.168591 5528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:48.168756 5528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "29d0d5ec9da54d148c47f51025cbf3cc"
format_stamp: "Formatted at 2026-08-06 04:39:48 on dist-test-slave-vkjp"
server_key: "8afa069670efc6db6461b4db5c8f9ce9"
server_key_iv: "c2d73953ad8d2981bc4b12ef34358d08"
server_key_version: "encryptionkey@0"
I20260806 04:39:48.168887 5528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:48.216476 5528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:48.217289 5528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:48.217533 5528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:48.225309 5528 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:44027
I20260806 04:39:48.225417 5597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:44027 every 8 connection(s)
I20260806 04:39:48.226413 5528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:48.229565 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 5528
I20260806 04:39:48.229703 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:48.230028 5599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:48.230032 32410 external_mini_cluster.cc:1428] Setting key a0d02cbc5ac5ecf14e4b9ef176a5b6c3
I20260806 04:39:48.237049 5599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc: Bootstrap starting.
I20260806 04:39:48.239837 5599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:48.240602 5599 log.cc:824] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:48.240610 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:48.231486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53152 (local address 127.31.166.190:44027)
0806 04:39:48.232162 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:48.232178 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:48.232215 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:39:48.233265 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:48.233285 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:48.233699 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:48.234010 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:48.234359 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.235516 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.236772 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.237373 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.237517 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:48.238513 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:48.238518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:48.238538 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:48.239009 (+ 471us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:48.239074 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:48.239176 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:48.239269 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:48.239441 (+ 172us) server_negotiation.cc:299] Negotiation successful
0806 04:39:48.239763 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":163,"threads_started":1}
I20260806 04:39:48.242919 5599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc: No bootstrap required, opened a new log
I20260806 04:39:48.243904 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:39:48.246903 5599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d0d5ec9da54d148c47f51025cbf3cc" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44027 } }
I20260806 04:39:48.247216 5599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:48.247303 5599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d0d5ec9da54d148c47f51025cbf3cc, State: Initialized, Role: FOLLOWER
I20260806 04:39:48.247761 5599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d0d5ec9da54d148c47f51025cbf3cc" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44027 } }
I20260806 04:39:48.247920 5599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:48.248023 5599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:48.248126 5599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:48.249080 5599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d0d5ec9da54d148c47f51025cbf3cc" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44027 } }
I20260806 04:39:48.249429 5599 leader_election.cc:302] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29d0d5ec9da54d148c47f51025cbf3cc; no voters:
I20260806 04:39:48.249751 5599 leader_election.cc:288] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:48.249936 5604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:48.250201 5604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [term 1 LEADER]: Becoming Leader. State: Replica: 29d0d5ec9da54d148c47f51025cbf3cc, State: Running, Role: LEADER
I20260806 04:39:48.250537 5604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d0d5ec9da54d148c47f51025cbf3cc" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44027 } }
I20260806 04:39:48.251050 5599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:48.252312 5605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29d0d5ec9da54d148c47f51025cbf3cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d0d5ec9da54d148c47f51025cbf3cc" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44027 } } }
I20260806 04:39:48.252466 5605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:48.252856 5606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29d0d5ec9da54d148c47f51025cbf3cc. Latest consensus state: current_term: 1 leader_uuid: "29d0d5ec9da54d148c47f51025cbf3cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29d0d5ec9da54d148c47f51025cbf3cc" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44027 } } }
I20260806 04:39:48.252975 5606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:48.253279 5616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:48.256491 5616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:48.261934 5616 catalog_manager.cc:1347] Generated new cluster ID: b3f0de03cbfb43b8833278f31921b9d3
I20260806 04:39:48.262015 5616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:48.272744 5616 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:48.274034 5616 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:48.283551 5616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29d0d5ec9da54d148c47f51025cbf3cc: Generated new TSK 0
I20260806 04:39:48.284281 5616 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 04:39:48.316756 5634 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 04:39:48.316865 5634 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 04:39:48.317654 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:48.292048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53158 (local address 127.31.166.190:44027)
0806 04:39:48.292198 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:48.292203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:48.292416 (+ 213us) server_negotiation.cc:407] Connection header received
0806 04:39:48.292617 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:48.292620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:48.292665 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:48.292781 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0806 04:39:48.293045 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.293582 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.294496 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.294718 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.294825 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:48.295222 (+ 397us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 04:39:48.317097 (+ 21875us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36343/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 04:39:48.317443 (+ 346us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53158: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36343/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 04:39:48.317740 5602 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53158: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36343/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 04:39:48.317838 5633 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:39:48.291964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:44027 (local address 127.0.0.1:53158)
0806 04:39:48.292313 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:48.292336 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:48.292467 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:48.292783 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:48.292786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:48.292795 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0806 04:39:48.292934 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:48.292939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.293718 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.293722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:48.294361 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:48.294369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.294843 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.294847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:48.294953 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:48.294977 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 04:39:48.317340 (+ 22363us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36343/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 04:39:48.317602 (+ 262us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.166.190:44027: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36343/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":128,"threads_started":1}
I20260806 04:39:48.318503 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 5528
2026-08-06T04:39:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (358 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 04:39:48.359335 32410 mini_oidc.cc:152] Starting JWKS server
I20260806 04:39:48.360249 32410 webserver.cc:460] Webserver started at https://127.0.0.1:36665/ using document root <none> and password file <none>
I20260806 04:39:48.360389 32410 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 04:39:48.360667 32410 webserver.cc:460] Webserver started at http://127.0.0.1:38441/ using document root <none> and password file <none>
2026-08-06T04:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:48Z Disabled control of system clock
I20260806 04:39:48.370414 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:36901
--webserver_interface=127.31.166.190
--webserver_port=0
--builtin_ntp_servers=127.31.166.148:37529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:36901
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36665/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 04:39:48.482872 5640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:48.483222 5640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:48.483268 5640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:48.487304 5640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:48.487362 5640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:48.487378 5640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:48.487396 5640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:48.487411 5640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 04:39:48.487426 5640 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 04:39:48.487439 5640 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36665/jwks.json
I20260806 04:39:48.492933 5640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:37529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:36901
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:36901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36665/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:48.494041 5640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:48.495502 5640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:48.504096 5647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:48.504096 5649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:48.504114 5645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:48.504196 5640 server_base.cc:1034] running on GCE node
I20260806 04:39:48.505015 5640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:48.506228 5640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:48.507431 5640 hybrid_clock.cc:648] HybridClock initialized: now 1785991188507415 us; error 35 us; skew 500 ppm
I20260806 04:39:48.509413 5640 webserver.cc:460] Webserver started at http://127.31.166.190:39435/ using document root <none> and password file <none>
I20260806 04:39:48.510012 5640 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:48.510108 5640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:48.510354 5640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:48.512473 5640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "1ff6580d2b2f44f388a96b60779760f7"
format_stamp: "Formatted at 2026-08-06 04:39:48 on dist-test-slave-vkjp"
server_key: "1da90016c70dd5903ad4128d2f966687"
server_key_iv: "69ae494b39bec00b76d5e96d4fe02354"
server_key_version: "encryptionkey@0"
I20260806 04:39:48.513064 5640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "1ff6580d2b2f44f388a96b60779760f7"
format_stamp: "Formatted at 2026-08-06 04:39:48 on dist-test-slave-vkjp"
server_key: "1da90016c70dd5903ad4128d2f966687"
server_key_iv: "69ae494b39bec00b76d5e96d4fe02354"
server_key_version: "encryptionkey@0"
I20260806 04:39:48.516969 5640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 04:39:48.519470 5655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:48.520704 5640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:39:48.520846 5640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "1ff6580d2b2f44f388a96b60779760f7"
format_stamp: "Formatted at 2026-08-06 04:39:48 on dist-test-slave-vkjp"
server_key: "1da90016c70dd5903ad4128d2f966687"
server_key_iv: "69ae494b39bec00b76d5e96d4fe02354"
server_key_version: "encryptionkey@0"
I20260806 04:39:48.520979 5640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:48.536095 5640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:48.536893 5640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:48.537109 5640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:48.544621 5640 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:36901
I20260806 04:39:48.544634 5708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:36901 every 8 connection(s)
I20260806 04:39:48.545627 5640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:48.546475 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 5640
I20260806 04:39:48.546626 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:48.546921 32410 external_mini_cluster.cc:1428] Setting key 37832a3ced27ffba10fe38a705bc4cad
I20260806 04:39:48.549292 5709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:48.556041 5709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7: Bootstrap starting.
I20260806 04:39:48.556653 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:48.548426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.31.166.190:36901)
0806 04:39:48.549097 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:48.549112 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:48.549145 (+ 33us) server_negotiation.cc:407] Connection header received
0806 04:39:48.549848 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:48.549871 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:48.550266 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:48.550550 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:48.551006 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.552032 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.553260 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.553709 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.553825 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:48.554774 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:48.554779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:48.554791 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:48.555254 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:48.555310 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:48.555331 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:48.555408 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:48.555575 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 04:39:48.555885 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":150,"threads_started":1}
I20260806 04:39:48.558569 5709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:48.559568 5709 log.cc:824] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:48.559924 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:39:48.561925 5709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7: No bootstrap required, opened a new log
I20260806 04:39:48.565574 5709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff6580d2b2f44f388a96b60779760f7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36901 } }
I20260806 04:39:48.565836 5709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:48.565907 5709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff6580d2b2f44f388a96b60779760f7, State: Initialized, Role: FOLLOWER
I20260806 04:39:48.566371 5709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff6580d2b2f44f388a96b60779760f7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36901 } }
I20260806 04:39:48.566522 5709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:48.566594 5709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:48.566702 5709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:48.567932 5709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff6580d2b2f44f388a96b60779760f7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36901 } }
I20260806 04:39:48.568284 5709 leader_election.cc:302] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ff6580d2b2f44f388a96b60779760f7; no voters:
I20260806 04:39:48.568607 5709 leader_election.cc:288] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:48.568764 5714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:48.569034 5714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [term 1 LEADER]: Becoming Leader. State: Replica: 1ff6580d2b2f44f388a96b60779760f7, State: Running, Role: LEADER
I20260806 04:39:48.569330 5714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff6580d2b2f44f388a96b60779760f7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36901 } }
I20260806 04:39:48.569875 5709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:48.571198 5716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff6580d2b2f44f388a96b60779760f7. Latest consensus state: current_term: 1 leader_uuid: "1ff6580d2b2f44f388a96b60779760f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff6580d2b2f44f388a96b60779760f7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36901 } } }
I20260806 04:39:48.571352 5716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:48.571581 5715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ff6580d2b2f44f388a96b60779760f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff6580d2b2f44f388a96b60779760f7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36901 } } }
I20260806 04:39:48.571693 5715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:48.571841 5724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:48.575282 5724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:48.581261 5724 catalog_manager.cc:1347] Generated new cluster ID: 90b80842053d4aa28391ae6caf0269ff
I20260806 04:39:48.581360 5724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:48.598810 5724 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:48.600145 5724 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:48.608035 5724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ff6580d2b2f44f388a96b60779760f7: Generated new TSK 0
I20260806 04:39:48.608943 5724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:39:48.628947 5712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:48.623608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.31.166.190:36901)
0806 04:39:48.623752 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:48.623757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:48.623918 (+ 161us) server_negotiation.cc:407] Connection header received
0806 04:39:48.624083 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:48.624086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:48.624136 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:48.624266 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0806 04:39:48.624523 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.625139 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.626433 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.626641 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.626723 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:48.627123 (+ 400us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 04:39:48.628586 (+ 1463us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 04:39:48.628810 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53300: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 04:39:48.629022 5712 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53300: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 04:39:48.630746 5740 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:39:48.623482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36901 (local address 127.0.0.1:53300)
0806 04:39:48.623804 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:48.623820 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:48.623949 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:48.624289 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:48.624292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:48.624300 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0806 04:39:48.624415 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:48.624421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.625248 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.625252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:48.626300 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:48.626311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.626770 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.626774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:48.626863 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:48.626910 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 04:39:48.628750 (+ 1840us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 04:39:48.630574 (+ 1824us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.166.190:36901: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36665/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":1618,"thread_start_us":118,"threads_started":1}
I20260806 04:39:48.631415 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 5640
E20260806 04:39:48.640846 5741 webserver.cc:533] Webserver: sslize failed:
E20260806 04:39:48.640923 5741 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-06T04:39:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (312 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T04:39:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:48Z Disabled control of system clock
I20260806 04:39:48.680786 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:44233
--webserver_interface=127.31.166.190
--webserver_port=0
--builtin_ntp_servers=127.31.166.148:36181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:44233
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/testchainca.cert with env {}
W20260806 04:39:48.792074 5745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:48.792344 5745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:48.792392 5745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:48.796522 5745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:48.796576 5745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:48.796592 5745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:48.796610 5745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:48.796625 5745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 04:39:48.796641 5745 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 04:39:48.796692 5745 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 04:39:48.802039 5745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:44233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:44233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:48.803107 5745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:48.804108 5745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:48.812645 5752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:48.812719 5751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:48.812708 5745 server_base.cc:1034] running on GCE node
W20260806 04:39:48.812706 5755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:48.813539 5745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:48.814642 5745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:48.815853 5745 hybrid_clock.cc:648] HybridClock initialized: now 1785991188815807 us; error 60 us; skew 500 ppm
I20260806 04:39:48.817837 5745 webserver.cc:460] Webserver started at http://127.31.166.190:36951/ using document root <none> and password file <none>
I20260806 04:39:48.818444 5745 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:48.818539 5745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:48.818773 5745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:48.820891 5745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "f2c10540521b411291f6c324886246b2"
format_stamp: "Formatted at 2026-08-06 04:39:48 on dist-test-slave-vkjp"
server_key: "feebae622c203434b6a2909e458830db"
server_key_iv: "e039d26785f59afb83dbabd5dceb4256"
server_key_version: "encryptionkey@0"
I20260806 04:39:48.821485 5745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "f2c10540521b411291f6c324886246b2"
format_stamp: "Formatted at 2026-08-06 04:39:48 on dist-test-slave-vkjp"
server_key: "feebae622c203434b6a2909e458830db"
server_key_iv: "e039d26785f59afb83dbabd5dceb4256"
server_key_version: "encryptionkey@0"
I20260806 04:39:48.825234 5745 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:39:48.827641 5762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:48.828928 5745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:48.829093 5745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "f2c10540521b411291f6c324886246b2"
format_stamp: "Formatted at 2026-08-06 04:39:48 on dist-test-slave-vkjp"
server_key: "feebae622c203434b6a2909e458830db"
server_key_iv: "e039d26785f59afb83dbabd5dceb4256"
server_key_version: "encryptionkey@0"
I20260806 04:39:48.829211 5745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:48.846805 5745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:48.847545 5745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:48.847754 5745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:48.855010 5745 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:44233
I20260806 04:39:48.855041 5818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:44233 every 8 connection(s)
I20260806 04:39:48.855998 5745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:48.856715 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 5745
I20260806 04:39:48.856858 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:48.857165 32410 external_mini_cluster.cc:1428] Setting key d4c18448060a1e1e9c88bab46fa21af1
I20260806 04:39:48.859731 5819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:48.865474 5819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2: Bootstrap starting.
I20260806 04:39:48.866619 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:48.858551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47040 (local address 127.31.166.190:44233)
0806 04:39:48.859405 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:48.859422 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:48.859460 (+ 38us) server_negotiation.cc:407] Connection header received
0806 04:39:48.860159 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:48.860176 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:48.860560 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:48.860905 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:48.861346 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.862309 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.863310 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.863777 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.863980 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:48.864930 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:48.864935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:48.864948 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:39:48.865322 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:48.865380 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:39:48.865401 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:48.865509 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:48.865682 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 04:39:48.865942 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":161,"threads_started":1}
I20260806 04:39:48.868044 5819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:48.868923 5819 log.cc:824] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:48.869576 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:39:48.871040 5819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2: No bootstrap required, opened a new log
I20260806 04:39:48.874660 5819 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c10540521b411291f6c324886246b2" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44233 } }
I20260806 04:39:48.874893 5819 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:48.874928 5819 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c10540521b411291f6c324886246b2, State: Initialized, Role: FOLLOWER
I20260806 04:39:48.875371 5819 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c10540521b411291f6c324886246b2" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44233 } }
I20260806 04:39:48.875507 5819 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:48.875557 5819 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:48.875633 5819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:48.876751 5819 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c10540521b411291f6c324886246b2" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44233 } }
I20260806 04:39:48.877090 5819 leader_election.cc:302] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2c10540521b411291f6c324886246b2; no voters:
I20260806 04:39:48.877411 5819 leader_election.cc:288] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:48.877560 5824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:48.877836 5824 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [term 1 LEADER]: Becoming Leader. State: Replica: f2c10540521b411291f6c324886246b2, State: Running, Role: LEADER
I20260806 04:39:48.878145 5824 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c10540521b411291f6c324886246b2" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44233 } }
I20260806 04:39:48.878664 5819 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:48.879917 5826 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2c10540521b411291f6c324886246b2. Latest consensus state: current_term: 1 leader_uuid: "f2c10540521b411291f6c324886246b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c10540521b411291f6c324886246b2" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44233 } } }
I20260806 04:39:48.880077 5826 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:48.880429 5825 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2c10540521b411291f6c324886246b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c10540521b411291f6c324886246b2" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44233 } } }
I20260806 04:39:48.880625 5825 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:48.880514 5833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:48.884124 5833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:48.890093 5833 catalog_manager.cc:1347] Generated new cluster ID: 6c081d078a264ab9a2dd99165ed7b775
I20260806 04:39:48.890173 5833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:48.927234 5833 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:48.928565 5833 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:48.935519 5833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2c10540521b411291f6c324886246b2: Generated new TSK 0
I20260806 04:39:48.936290 5833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:39:48.964318 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:48.945808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47050 (local address 127.31.166.190:44233)
0806 04:39:48.945940 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:48.945945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:48.946176 (+ 231us) server_negotiation.cc:407] Connection header received
0806 04:39:48.946306 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:48.946309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:48.946367 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:48.946490 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0806 04:39:48.946695 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.947517 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.948435 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.948788 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.948899 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:48.949210 (+ 311us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 04:39:48.963824 (+ 14614us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 04:39:48.964161 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47050: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 04:39:48.964411 5822 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47050: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 04:39:48.964854 5849 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:39:48.945732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:44233 (local address 127.0.0.1:47050)
0806 04:39:48.946089 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:48.946102 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:48.946199 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:48.946493 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:48.946495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:48.946502 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0806 04:39:48.946594 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:48.946598 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.947674 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.947677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:48.948307 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:48.948315 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:48.948932 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:48.948935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:48.949005 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:48.949030 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 04:39:48.964099 (+ 15069us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 04:39:48.964279 (+ 180us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.166.190:44233: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":177,"threads_started":1}
I20260806 04:39:48.965574 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 5745
2026-08-06T04:39:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (313 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (124 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (121 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:39:49 dist-test-slave-vkjp krb5kdc[5862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:39:49 dist-test-slave-vkjp krb5kdc[5862](info): set up 2 sockets
Aug 06 04:39:49 dist-test-slave-vkjp krb5kdc[5862](info): commencing operation
krb5kdc: starting...
W20260806 04:39:50.290283 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:39:50 dist-test-slave-vkjp krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:39:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:39:50.414258 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:39579
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:39579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:50.523005 5877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:50.523358 5877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:50.523414 5877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:50.527416 5877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:50.527472 5877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:50.527491 5877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:50.527508 5877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:50.527524 5877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:39:50.532989 5877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:39579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:39579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:50.533959 5877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:50.534930 5877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:50.543502 5882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:50.543598 5886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:50.543550 5883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:50.543722 5877 server_base.cc:1034] running on GCE node
I20260806 04:39:50.544494 5877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:50.545694 5877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:50.546893 5877 hybrid_clock.cc:648] HybridClock initialized: now 1785991190546863 us; error 42 us; skew 500 ppm
Aug 06 04:39:50 dist-test-slave-vkjp krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991190, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:50.553023 5877 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:39:50.553956 5877 webserver.cc:460] Webserver started at http://127.31.166.190:40343/ using document root <none> and password file <none>
I20260806 04:39:50.554466 5877 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:50.554522 5877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:50.554682 5877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:50.556680 5877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "b66895831a4d4aeca912bf0ff59981e7"
format_stamp: "Formatted at 2026-08-06 04:39:50 on dist-test-slave-vkjp"
server_key: "e8378759621d1ed1d6945e46c5e87c66"
server_key_iv: "60c0604f77165f60a321de3258018b82"
server_key_version: "encryptionkey@0"
I20260806 04:39:50.557214 5877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "b66895831a4d4aeca912bf0ff59981e7"
format_stamp: "Formatted at 2026-08-06 04:39:50 on dist-test-slave-vkjp"
server_key: "e8378759621d1ed1d6945e46c5e87c66"
server_key_iv: "60c0604f77165f60a321de3258018b82"
server_key_version: "encryptionkey@0"
I20260806 04:39:50.560757 5877 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260806 04:39:50.563015 5894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:50.564195 5877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:39:50.564323 5877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "b66895831a4d4aeca912bf0ff59981e7"
format_stamp: "Formatted at 2026-08-06 04:39:50 on dist-test-slave-vkjp"
server_key: "e8378759621d1ed1d6945e46c5e87c66"
server_key_iv: "60c0604f77165f60a321de3258018b82"
server_key_version: "encryptionkey@0"
I20260806 04:39:50.564422 5877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:50.575352 5877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:50.578351 5877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:50.578547 5877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:50.585527 5877 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:39579
I20260806 04:39:50.585539 5952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:39579 every 8 connection(s)
I20260806 04:39:50.586489 5877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:50.589187 5953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:50.590154 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 5877
I20260806 04:39:50.590281 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:50.590619 32410 external_mini_cluster.cc:1428] Setting key c21dad73483734fbfcbe746cefc2564c
I20260806 04:39:50.595609 5953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7: Bootstrap starting.
I20260806 04:39:50.598242 5953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:50.599244 5953 log.cc:824] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:39:50 dist-test-slave-vkjp krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:50.601513 5953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7: No bootstrap required, opened a new log
I20260806 04:39:50.605769 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:50.592024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.31.166.190:39579)
0806 04:39:50.592433 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:50.592448 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:50.592484 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:39:50.593098 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:50.593113 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:50.593426 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:50.593687 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:50.594587 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:50.595609 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:50.596782 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:50.597252 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:50.597371 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:50.600572 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:50.600578 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:50.600594 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:50.600625 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:50.602538 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:50.602966 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:50.602976 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:50.602982 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:50.603161 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:50.603467 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:50.603470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:50.603472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:50.604066 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:50.604274 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:50.604524 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 04:39:50.604819 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":126,"threads_started":1}
I20260806 04:39:50.606158 5953 raft_consensus.cc:348] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66895831a4d4aeca912bf0ff59981e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39579 } }
I20260806 04:39:50.606431 5953 raft_consensus.cc:374] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:50.606480 5953 raft_consensus.cc:729] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66895831a4d4aeca912bf0ff59981e7, State: Initialized, Role: FOLLOWER
I20260806 04:39:50.606950 5953 consensus_queue.cc:260] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66895831a4d4aeca912bf0ff59981e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39579 } }
I20260806 04:39:50.607131 5953 raft_consensus.cc:388] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:50.607206 5953 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:50.607331 5953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:50.608430 5953 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66895831a4d4aeca912bf0ff59981e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39579 } }
I20260806 04:39:50.608764 5953 leader_election.cc:302] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b66895831a4d4aeca912bf0ff59981e7; no voters:
I20260806 04:39:50.609088 5953 leader_election.cc:288] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:50.609248 5958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:50.609460 5958 raft_consensus.cc:686] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [term 1 LEADER]: Becoming Leader. State: Replica: b66895831a4d4aeca912bf0ff59981e7, State: Running, Role: LEADER
I20260806 04:39:50.609768 5958 consensus_queue.cc:237] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66895831a4d4aeca912bf0ff59981e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39579 } }
I20260806 04:39:50.610294 5953 sys_catalog.cc:564] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:50.611449 5959 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b66895831a4d4aeca912bf0ff59981e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66895831a4d4aeca912bf0ff59981e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39579 } } }
I20260806 04:39:50.611581 5959 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:50.611747 5960 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66895831a4d4aeca912bf0ff59981e7. Latest consensus state: current_term: 1 leader_uuid: "b66895831a4d4aeca912bf0ff59981e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66895831a4d4aeca912bf0ff59981e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39579 } } }
I20260806 04:39:50.611944 5960 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:50.612169 5967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:50.614715 5967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:50.619867 5967 catalog_manager.cc:1347] Generated new cluster ID: 7f10704efdef470d90a84439f9f2fb80
I20260806 04:39:50.619935 5967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:50.635519 5967 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:50.636837 5967 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:50.646049 5967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b66895831a4d4aeca912bf0ff59981e7: Generated new TSK 0
I20260806 04:39:50.646821 5967 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:39:50.687654 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39579
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:50.797015 5980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:50.797259 5980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:50.797308 5980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:50.801337 5980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:50.801393 5980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:50.801484 5980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:39:50.806692 5980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:50.807679 5980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:50.808650 5980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:50.818553 5986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:50.818562 5988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:50.818562 5985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:50.819032 5980 server_base.cc:1034] running on GCE node
I20260806 04:39:50.819554 5980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:50.820225 5980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:50.821422 5980 hybrid_clock.cc:648] HybridClock initialized: now 1785991190821402 us; error 35 us; skew 500 ppm
Aug 06 04:39:50 dist-test-slave-vkjp krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991190, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:50.824524 5980 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:39:50.825402 5980 webserver.cc:460] Webserver started at http://127.31.166.129:45981/ using document root <none> and password file <none>
I20260806 04:39:50.825969 5980 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:50.826058 5980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:50.826298 5980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:50.828454 5980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "b66cacef389f47aab6edfacc5a6e6d5b"
format_stamp: "Formatted at 2026-08-06 04:39:50 on dist-test-slave-vkjp"
server_key: "87504a9aa76f5feaa6af979fa1e4569b"
server_key_iv: "1c515a0f1a3163873e09ca4134f75789"
server_key_version: "encryptionkey@0"
I20260806 04:39:50.829030 5980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "b66cacef389f47aab6edfacc5a6e6d5b"
format_stamp: "Formatted at 2026-08-06 04:39:50 on dist-test-slave-vkjp"
server_key: "87504a9aa76f5feaa6af979fa1e4569b"
server_key_iv: "1c515a0f1a3163873e09ca4134f75789"
server_key_version: "encryptionkey@0"
I20260806 04:39:50.833222 5980 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:39:50.836079 5996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:50.837461 5980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 04:39:50.837606 5980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "b66cacef389f47aab6edfacc5a6e6d5b"
format_stamp: "Formatted at 2026-08-06 04:39:50 on dist-test-slave-vkjp"
server_key: "87504a9aa76f5feaa6af979fa1e4569b"
server_key_iv: "1c515a0f1a3163873e09ca4134f75789"
server_key_version: "encryptionkey@0"
I20260806 04:39:50.837733 5980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:50.854885 5980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:50.858484 5980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:50.858717 5980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:50.859426 5980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:50.860333 5980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:50.860411 5980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:50.860491 5980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:50.860538 5980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:50.870570 5980 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:36411
I20260806 04:39:50.870592 6122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:36411 every 8 connection(s)
I20260806 04:39:50.871646 5980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:39:50.873921 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 5980
I20260806 04:39:50.874022 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:39:50.874312 32410 external_mini_cluster.cc:1428] Setting key ad7a60b08d4575c08c85bdb58bce7cb1
Aug 06 04:39:50 dist-test-slave-vkjp krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991190, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
I20260806 04:39:50.887132 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:50.873608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:48629 (local address 127.31.166.190:39579)
0806 04:39:50.873783 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:50.873788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:50.874809 (+ 1021us) server_negotiation.cc:407] Connection header received
0806 04:39:50.876405 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:50.876411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:50.876516 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:50.876605 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:50.878001 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:50.878527 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:50.879826 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:50.880279 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:50.880401 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:50.883159 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:50.883166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:50.883169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:50.883203 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:50.884879 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:50.885491 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:50.885495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:50.885496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:50.885576 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:50.885935 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:50.885940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:50.885942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:50.886151 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:50.886262 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:50.886771 (+ 509us) server_negotiation.cc:299] Negotiation successful
0806 04:39:50.886918 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 04:39:50.887960 6125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:50.874125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39579 (local address 127.31.166.129:48629)
0806 04:39:50.874637 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:50.874690 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:50.876177 (+ 1487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:50.876762 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:50.876768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:50.877110 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:50.877836 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:50.877859 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:50.878658 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:50.878664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:50.879691 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:50.879701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:50.880432 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:50.880435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:50.880570 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:50.881221 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:50.881237 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:50.882950 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:50.885001 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:50.885007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:50.885021 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:50.885354 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:50.885681 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:50.885688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:50.885690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:50.885838 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:50.886264 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:50.886274 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:50.886535 (+ 261us) client_negotiation.cc:769] Sending connection context
0806 04:39:50.886731 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 04:39:50.887076 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":146,"threads_started":1}
I20260806 04:39:50.889187 6123 heartbeater.cc:347] Connected to a master server at 127.31.166.190:39579
I20260806 04:39:50.889477 6123 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:50.890014 6123 heartbeater.cc:511] Master 127.31.166.190:39579 requested a full tablet report, sending...
I20260806 04:39:50.891569 5913 ts_manager.cc:194] Registered new tserver with Master: b66cacef389f47aab6edfacc5a6e6d5b (127.31.166.129:36411)
I20260806 04:39:50.892784 5913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:48629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:39:50.905887 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39579
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:51.020351 6128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:51.020607 6128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:51.020650 6128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:51.024758 6128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:51.024811 6128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:51.024900 6128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:39:51.030157 6128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:51.031216 6128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:51.032204 6128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:51.041836 6138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:51.041846 6134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:51.041882 6135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:51.042307 6128 server_base.cc:1034] running on GCE node
I20260806 04:39:51.042670 6128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:51.043296 6128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:51.044458 6128 hybrid_clock.cc:648] HybridClock initialized: now 1785991191044441 us; error 29 us; skew 500 ppm
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991191, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:51.047164 6128 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:39:51.047993 6128 webserver.cc:460] Webserver started at http://127.31.166.130:37371/ using document root <none> and password file <none>
I20260806 04:39:51.048533 6128 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:51.048623 6128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:51.048856 6128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:51.050813 6128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "aa72433460004e53af0bed89c9d9e34a"
format_stamp: "Formatted at 2026-08-06 04:39:51 on dist-test-slave-vkjp"
server_key: "5d6e959cafa151494027bedcba39a3d8"
server_key_iv: "c92116768ba71bbbebd374cad9092a34"
server_key_version: "encryptionkey@0"
I20260806 04:39:51.051390 6128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "aa72433460004e53af0bed89c9d9e34a"
format_stamp: "Formatted at 2026-08-06 04:39:51 on dist-test-slave-vkjp"
server_key: "5d6e959cafa151494027bedcba39a3d8"
server_key_iv: "c92116768ba71bbbebd374cad9092a34"
server_key_version: "encryptionkey@0"
I20260806 04:39:51.054939 6128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:39:51.057296 6147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:51.058427 6128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:39:51.058573 6128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "aa72433460004e53af0bed89c9d9e34a"
format_stamp: "Formatted at 2026-08-06 04:39:51 on dist-test-slave-vkjp"
server_key: "5d6e959cafa151494027bedcba39a3d8"
server_key_iv: "c92116768ba71bbbebd374cad9092a34"
server_key_version: "encryptionkey@0"
I20260806 04:39:51.058679 6128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:51.074316 6128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:51.077136 6128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:51.077339 6128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:51.077910 6128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:51.078703 6128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:51.078783 6128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:51.078866 6128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:51.078920 6128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:51.089041 6128 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:36799
I20260806 04:39:51.089071 6271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:36799 every 8 connection(s)
I20260806 04:39:51.089917 6128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:39:51.092772 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 6128
I20260806 04:39:51.092931 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:39:51.093238 32410 external_mini_cluster.cc:1428] Setting key 7744bfb6858b7b636a0d94f6901389f2
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991191, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:39:51.105208 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:51.091719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:34743 (local address 127.31.166.190:39579)
0806 04:39:51.091862 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:51.091882 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:51.092609 (+ 727us) server_negotiation.cc:407] Connection header received
0806 04:39:51.094085 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:51.094088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:51.094147 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:51.094252 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:51.095741 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:51.096607 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:51.097898 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:51.098113 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:51.098197 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:51.101323 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:51.101326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:51.101328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:51.101351 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:51.102845 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:51.103443 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:51.103449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:51.103455 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:51.103537 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:51.103897 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:51.103945 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:51.103946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:51.104226 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:51.104349 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:51.104851 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 04:39:51.104985 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
I20260806 04:39:51.106163 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:51.092024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39579 (local address 127.31.166.130:34743)
0806 04:39:51.092455 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:51.092496 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:51.093878 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:51.094419 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:51.094426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:51.094811 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:51.095555 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:51.095577 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:51.096774 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:51.096776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:51.097760 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:51.097775 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:51.098270 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:51.098277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:51.098516 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:51.099330 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:51.099359 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:51.101141 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:51.102978 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:51.102984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:51.102998 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:51.103332 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:51.103633 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:51.103638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:51.103640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:51.103788 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:51.104347 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:51.104353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:51.104597 (+ 244us) client_negotiation.cc:769] Sending connection context
0806 04:39:51.104791 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 04:39:51.105119 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260806 04:39:51.107555 6272 heartbeater.cc:347] Connected to a master server at 127.31.166.190:39579
I20260806 04:39:51.107836 6272 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:51.108470 6272 heartbeater.cc:511] Master 127.31.166.190:39579 requested a full tablet report, sending...
I20260806 04:39:51.109779 5913 ts_manager.cc:194] Registered new tserver with Master: aa72433460004e53af0bed89c9d9e34a (127.31.166.130:36799)
I20260806 04:39:51.110708 5913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:34743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:39:51.122974 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39579
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:51.233541 6277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:51.233875 6277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:51.233920 6277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:51.237962 6277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:51.238016 6277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:51.238102 6277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:39:51.243637 6277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:51.244663 6277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:51.245658 6277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:51.255453 6284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:51.255400 6283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:51.255437 6277 server_base.cc:1034] running on GCE node
W20260806 04:39:51.255400 6286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:51.256054 6277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:51.256681 6277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:51.257861 6277 hybrid_clock.cc:648] HybridClock initialized: now 1785991191257811 us; error 63 us; skew 500 ppm
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991191, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:51.260725 6277 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:39:51.261674 6277 webserver.cc:460] Webserver started at http://127.31.166.131:44957/ using document root <none> and password file <none>
I20260806 04:39:51.262256 6277 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:51.262391 6277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:51.262594 6277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:51.264647 6277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "18f269a451bf4754b77c92975aa2fbb2"
format_stamp: "Formatted at 2026-08-06 04:39:51 on dist-test-slave-vkjp"
server_key: "d3efa3da8e7577b713073764594afdef"
server_key_iv: "915053d18e259562fc3f88d977e5ea74"
server_key_version: "encryptionkey@0"
I20260806 04:39:51.265158 6277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "18f269a451bf4754b77c92975aa2fbb2"
format_stamp: "Formatted at 2026-08-06 04:39:51 on dist-test-slave-vkjp"
server_key: "d3efa3da8e7577b713073764594afdef"
server_key_iv: "915053d18e259562fc3f88d977e5ea74"
server_key_version: "encryptionkey@0"
I20260806 04:39:51.268965 6277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 04:39:51.271360 6297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:51.272619 6277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:39:51.272739 6277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "18f269a451bf4754b77c92975aa2fbb2"
format_stamp: "Formatted at 2026-08-06 04:39:51 on dist-test-slave-vkjp"
server_key: "d3efa3da8e7577b713073764594afdef"
server_key_iv: "915053d18e259562fc3f88d977e5ea74"
server_key_version: "encryptionkey@0"
I20260806 04:39:51.272845 6277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:51.282176 6277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:51.285028 6277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:51.285238 6277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:51.285800 6277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:51.286612 6277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:51.286660 6277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:51.286731 6277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:51.286772 6277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:51.296015 6277 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:38101
I20260806 04:39:51.296099 6418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:38101 every 8 connection(s)
I20260806 04:39:51.296947 6277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:39:51.299348 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 6277
I20260806 04:39:51.299503 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:39:51.299925 32410 external_mini_cluster.cc:1428] Setting key f9c589f0a45f5d9d392d1d4e7360d7c5
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991191, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:39:51.312230 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:51.298914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:51689 (local address 127.31.166.190:39579)
0806 04:39:51.299070 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:51.299100 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:51.300185 (+ 1085us) server_negotiation.cc:407] Connection header received
0806 04:39:51.302108 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:51.302111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:51.302165 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:51.302249 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:51.303570 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:51.304105 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:51.305346 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:51.305559 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:51.305644 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:51.308266 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:51.308270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:51.308273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:51.308304 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:51.309674 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:51.310322 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:51.310324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:51.310326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:51.310414 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:51.310755 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:51.310758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:51.310759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:51.311178 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:51.311265 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:51.311883 (+ 618us) server_negotiation.cc:299] Negotiation successful
0806 04:39:51.312002 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:39:51.313115 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:51.299361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39579 (local address 127.31.166.131:51689)
0806 04:39:51.299981 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:51.300054 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:51.301867 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:51.302451 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:51.302459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:51.302846 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:51.303430 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:51.303445 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:51.304227 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:51.304229 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:51.305213 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:51.305223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:51.305783 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:51.305787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:51.305949 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:51.306503 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:51.306519 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:51.308078 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:51.309780 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:51.309786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:51.309799 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:51.310160 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:51.310498 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:51.310500 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:51.310501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:51.310636 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:51.311275 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:51.311280 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:51.311548 (+ 268us) client_negotiation.cc:769] Sending connection context
0806 04:39:51.311779 (+ 231us) client_negotiation.cc:240] Negotiation successful
0806 04:39:51.312061 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":170,"threads_started":1}
I20260806 04:39:51.314291 6419 heartbeater.cc:347] Connected to a master server at 127.31.166.190:39579
I20260806 04:39:51.314539 6419 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:51.315047 6419 heartbeater.cc:511] Master 127.31.166.190:39579 requested a full tablet report, sending...
I20260806 04:39:51.316170 5913 ts_manager.cc:194] Registered new tserver with Master: 18f269a451bf4754b77c92975aa2fbb2 (127.31.166.131:38101)
I20260806 04:39:51.316756 5913 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:51689
I20260806 04:39:51.326682 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/test-admin.keytab.
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:51.344586 32410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 04:39:51.347517 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:39:51.346014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.31.166.129:36411)
0806 04:39:51.346302 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:51.346317 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:51.346341 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:39:51.346477 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:51.346482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:51.346668 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:51.346788 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:51.347271 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45898: BlockingRecv error: recv got EOF from 127.0.0.1:45898 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
W20260806 04:39:51.347589 32410 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/krb5cc' of type FILE
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:51.348677 32410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:51.349670 32410 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.129@KRBTEST.COM
I20260806 04:39:51.359477 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:51.350550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45910 (local address 127.31.166.129:36411)
0806 04:39:51.350671 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:51.350675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:51.350886 (+ 211us) server_negotiation.cc:407] Connection header received
0806 04:39:51.350971 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:51.350974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:51.351015 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:51.351137 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:51.351616 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:51.352394 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:51.353199 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:51.353573 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:51.353655 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:51.355893 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:51.355902 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:51.355915 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:51.355937 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:51.357403 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:51.357848 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:51.357854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:51.357856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:51.357959 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:51.358297 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:51.358299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:51.358300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:51.358586 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:51.358783 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:51.359002 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 04:39:51.359315 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260806 04:39:51.363730 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:39:51.362691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45924 (local address 127.31.166.129:36411)
0806 04:39:51.362868 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:51.362872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:51.362912 (+ 40us) server_negotiation.cc:407] Connection header received
0806 04:39:51.363022 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:51.363024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:51.363060 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:51.363210 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:51.363640 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45924: BlockingRecv error: recv got EOF from 127.0.0.1:45924 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260806 04:39:51.363940 32410 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/krb5cc' of type FILE
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:51.365020 32410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:51.365928 32410 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.129@KRBTEST.COM
I20260806 04:39:51.374679 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:51.366808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.31.166.129:36411)
0806 04:39:51.366955 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:51.366959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:51.367035 (+ 76us) server_negotiation.cc:407] Connection header received
0806 04:39:51.367187 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:51.367189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:51.367225 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:51.367291 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:51.367787 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:51.368319 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:51.369095 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:51.369327 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:51.369430 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:51.371673 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:51.371676 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:51.371678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:51.371706 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:51.373113 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:51.373548 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:51.373555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:51.373556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:51.373659 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:51.373909 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:51.373911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:51.373913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:51.374113 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:51.374232 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:51.374434 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 04:39:51.374546 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 04:39:51.377843 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:39:51.376757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45936 (local address 127.31.166.129:36411)
0806 04:39:51.376906 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:51.376911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:51.377035 (+ 124us) server_negotiation.cc:407] Connection header received
0806 04:39:51.377176 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:51.377179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:51.377225 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:51.377308 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:51.377719 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45936: BlockingRecv error: recv got EOF from 127.0.0.1:45936 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260806 04:39:51.378070 32410 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785991160137184-32410-0/krb5kdc/krb5cc' of type FILE
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:51.379251 32410 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[5862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:51.380214 32410 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[5862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.129@KRBTEST.COM
I20260806 04:39:51.390344 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:51.381078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.31.166.129:36411)
0806 04:39:51.381199 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:51.381203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:51.381378 (+ 175us) server_negotiation.cc:407] Connection header received
0806 04:39:51.381461 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:51.381463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:51.381498 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:51.381562 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:51.382481 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:51.383402 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:51.384187 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:51.384413 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:51.384493 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:51.387190 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:51.387193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:51.387196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:51.387220 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:51.388659 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:51.389156 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:51.389158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:51.389159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:51.389230 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:51.389507 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:51.389509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:51.389510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:51.389663 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:51.389760 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:51.390029 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 04:39:51.390144 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:39:51.391491 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 5980
I20260806 04:39:51.401595 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 6128
I20260806 04:39:51.407438 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 6277
I20260806 04:39:51.414108 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 5877
2026-08-06T04:39:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2205 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[6441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[6441](info): set up 2 sockets
Aug 06 04:39:51 dist-test-slave-vkjp krb5kdc[6441](info): commencing operation
krb5kdc: starting...
W20260806 04:39:52.494441 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:39:52 dist-test-slave-vkjp krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:39:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.190.keytab.
Entry for principal oryx/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.190.keytab.
I20260806 04:39:52.623029 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:43117
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.190.keytab
--principal=oryx/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:43117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:52.732939 6455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:52.733253 6455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:52.733336 6455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:52.737385 6455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:52.737493 6455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:52.737531 6455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:52.737587 6455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:52.737622 6455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:39:52.743029 6455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:43117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.190.keytab
--principal=oryx/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:43117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:52.744200 6455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:52.745260 6455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:52.754106 6464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:52.754112 6461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:52.754325 6455 server_base.cc:1034] running on GCE node
W20260806 04:39:52.754106 6460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:52.755028 6455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:52.756230 6455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:52.757445 6455 hybrid_clock.cc:648] HybridClock initialized: now 1785991192757421 us; error 42 us; skew 500 ppm
Aug 06 04:39:52 dist-test-slave-vkjp krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991192, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:52.760404 6455 init.cc:377] Logged in from keytab as oryx/127.31.166.190@KRBTEST.COM (short username oryx)
I20260806 04:39:52.761307 6455 webserver.cc:460] Webserver started at http://127.31.166.190:44563/ using document root <none> and password file <none>
I20260806 04:39:52.761839 6455 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:52.761902 6455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:52.762069 6455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:52.764235 6455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "824064c26018499f879fdf4970b0b8e4"
format_stamp: "Formatted at 2026-08-06 04:39:52 on dist-test-slave-vkjp"
server_key: "89136d722c0c1267849a6407cd175a19"
server_key_iv: "0c2e5f89d70344d3a435896b941225bd"
server_key_version: "encryptionkey@0"
I20260806 04:39:52.764767 6455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "824064c26018499f879fdf4970b0b8e4"
format_stamp: "Formatted at 2026-08-06 04:39:52 on dist-test-slave-vkjp"
server_key: "89136d722c0c1267849a6407cd175a19"
server_key_iv: "0c2e5f89d70344d3a435896b941225bd"
server_key_version: "encryptionkey@0"
I20260806 04:39:52.769454 6455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:39:52.772028 6471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:52.773389 6455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:39:52.773494 6455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "824064c26018499f879fdf4970b0b8e4"
format_stamp: "Formatted at 2026-08-06 04:39:52 on dist-test-slave-vkjp"
server_key: "89136d722c0c1267849a6407cd175a19"
server_key_iv: "0c2e5f89d70344d3a435896b941225bd"
server_key_version: "encryptionkey@0"
I20260806 04:39:52.773588 6455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:52.794862 6455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:52.798261 6455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:52.798429 6455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:52.805696 6455 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:43117
I20260806 04:39:52.806008 6526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:43117 every 8 connection(s)
I20260806 04:39:52.806686 6455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:39:52.809365 6527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:52.809912 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 6455
I20260806 04:39:52.810053 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:39:52.810412 32410 external_mini_cluster.cc:1428] Setting key a33947580626384daeb04e2de73d7033
I20260806 04:39:52.815800 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4: Bootstrap starting.
I20260806 04:39:52.818465 6527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:52.819384 6527 log.cc:824] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:39:52 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
I20260806 04:39:52.821851 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4: No bootstrap required, opened a new log
I20260806 04:39:52.825671 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:52.811822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50174 (local address 127.31.166.190:43117)
0806 04:39:52.812239 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:52.812263 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:52.812316 (+ 53us) server_negotiation.cc:407] Connection header received
0806 04:39:52.812991 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:52.813008 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:52.813318 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:52.813592 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:52.814498 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:52.815629 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:52.816784 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:52.817240 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:52.817372 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:52.820392 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:52.820398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:52.820413 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:52.820441 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:52.822372 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:52.822873 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:52.822879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:52.822884 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:52.823016 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:52.823376 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:52.823379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:52.823381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:52.823931 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:52.824167 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:52.824380 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 04:39:52.824706 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":149,"threads_started":1}
I20260806 04:39:52.826435 6527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } }
I20260806 04:39:52.826702 6527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:52.826817 6527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 824064c26018499f879fdf4970b0b8e4, State: Initialized, Role: FOLLOWER
I20260806 04:39:52.827286 6527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } }
I20260806 04:39:52.827412 6527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:39:52.827498 6527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:39:52.827595 6527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:52.828650 6527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } }
I20260806 04:39:52.828989 6527 leader_election.cc:302] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 824064c26018499f879fdf4970b0b8e4; no voters:
I20260806 04:39:52.829301 6527 leader_election.cc:288] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:39:52.829452 6533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:52.829694 6533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [term 1 LEADER]: Becoming Leader. State: Replica: 824064c26018499f879fdf4970b0b8e4, State: Running, Role: LEADER
I20260806 04:39:52.830029 6533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } }
I20260806 04:39:52.830525 6527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:52.831718 6535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 824064c26018499f879fdf4970b0b8e4. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } }
I20260806 04:39:52.831857 6535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:52.832159 6534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } }
I20260806 04:39:52.832320 6534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:52.832226 6544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:52.835583 6544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:52.841121 6544 catalog_manager.cc:1347] Generated new cluster ID: b20d78810fde47c7ae5a4e7969b72dc4
I20260806 04:39:52.841189 6544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:52.855952 6544 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:39:52.856910 6544 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:52.865422 6544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4: Generated new TSK 0
I20260806 04:39:52.866101 6544 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.129.keytab.
Entry for principal oryx/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.129.keytab.
I20260806 04:39:52.899430 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43117
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.129.keytab
--principal=oryx/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:53.013798 6558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:53.014068 6558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:53.014113 6558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:53.018621 6558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:53.018697 6558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:53.018798 6558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:39:53.024300 6558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.129.keytab
--principal=oryx/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:53.025458 6558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:53.026486 6558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:53.036530 6563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:53.036721 6558 server_base.cc:1034] running on GCE node
W20260806 04:39:53.036612 6564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:53.036531 6566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:53.037225 6558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:53.037922 6558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:53.039162 6558 hybrid_clock.cc:648] HybridClock initialized: now 1785991193039147 us; error 30 us; skew 500 ppm
Aug 06 04:39:53 dist-test-slave-vkjp krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991193, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:53.041940 6558 init.cc:377] Logged in from keytab as oryx/127.31.166.129@KRBTEST.COM (short username oryx)
I20260806 04:39:53.042785 6558 webserver.cc:460] Webserver started at http://127.31.166.129:46163/ using document root <none> and password file <none>
I20260806 04:39:53.043421 6558 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:53.043514 6558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:53.043748 6558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:53.045825 6558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "0c2609e68bce4b6c8edc78b221bd9f38"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "acd4a081271ee7028d74f78df48df91f"
server_key_iv: "326e615f3aaef6438739cf5ceeec3e28"
server_key_version: "encryptionkey@0"
I20260806 04:39:53.046414 6558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "0c2609e68bce4b6c8edc78b221bd9f38"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "acd4a081271ee7028d74f78df48df91f"
server_key_iv: "326e615f3aaef6438739cf5ceeec3e28"
server_key_version: "encryptionkey@0"
I20260806 04:39:53.050206 6558 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260806 04:39:53.052692 6574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:53.053953 6558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:39:53.054088 6558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "0c2609e68bce4b6c8edc78b221bd9f38"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "acd4a081271ee7028d74f78df48df91f"
server_key_iv: "326e615f3aaef6438739cf5ceeec3e28"
server_key_version: "encryptionkey@0"
I20260806 04:39:53.054201 6558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:53.068715 6558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:53.072319 6558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:53.072526 6558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:53.073125 6558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:53.073982 6558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:53.074084 6558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:53.074191 6558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:53.074224 6558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:53.083643 6558 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:39873
I20260806 04:39:53.083669 6697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:39873 every 8 connection(s)
I20260806 04:39:53.084628 6558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:39:53.086123 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 6558
I20260806 04:39:53.086289 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:39:53.086689 32410 external_mini_cluster.cc:1428] Setting key 86fe8aab0d34cd28a75edda7dea7d335
Aug 06 04:39:53 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991193, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.31.166.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.166.130@KRBTEST.COM" created.
I20260806 04:39:53.101415 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.086592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:46495 (local address 127.31.166.190:43117)
0806 04:39:53.086757 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:53.086762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:53.087847 (+ 1085us) server_negotiation.cc:407] Connection header received
0806 04:39:53.089906 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:53.089909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:53.089955 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:53.090028 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:53.091408 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.092197 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.093834 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.094318 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.094456 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.097336 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:53.097343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:53.097348 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:53.097381 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:53.098818 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.099576 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.099579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.099580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.099660 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.100096 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.100103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.100107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.100348 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:53.100433 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:53.101170 (+ 737us) server_negotiation.cc:299] Negotiation successful
0806 04:39:53.101285 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 04:39:53.102475 6700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.087048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.129:46495)
0806 04:39:53.087695 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:53.087741 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:53.089653 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:53.090254 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:53.090266 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:53.090661 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:53.091213 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.091229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.092360 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.092376 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.093637 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.093658 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.094540 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.094547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.094748 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.095479 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:53.095496 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:53.097153 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:53.098952 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.098957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.098970 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.099485 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.099751 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.099754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.099755 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.099929 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.100474 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:53.100482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:53.100872 (+ 390us) client_negotiation.cc:769] Sending connection context
0806 04:39:53.101118 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 04:39:53.101517 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":161,"threads_started":1}
I20260806 04:39:53.103907 6698 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43117
I20260806 04:39:53.104183 6698 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:53.104712 6698 heartbeater.cc:511] Master 127.31.166.190:43117 requested a full tablet report, sending...
I20260806 04:39:53.106417 6488 ts_manager.cc:194] Registered new tserver with Master: 0c2609e68bce4b6c8edc78b221bd9f38 (127.31.166.129:39873)
I20260806 04:39:53.107649 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:46495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.130.keytab.
Entry for principal oryx/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.130.keytab.
I20260806 04:39:53.118505 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43117
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.130.keytab
--principal=oryx/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:53.233618 6703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:53.233893 6703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:53.233968 6703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:53.237989 6703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:53.238042 6703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:53.238161 6703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:39:53.243438 6703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.130.keytab
--principal=oryx/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:53.244468 6703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:53.245604 6703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:53.255182 6712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:53.255218 6709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:53.255182 6710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:53.255359 6703 server_base.cc:1034] running on GCE node
I20260806 04:39:53.255853 6703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:53.256558 6703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:53.257750 6703 hybrid_clock.cc:648] HybridClock initialized: now 1785991193257724 us; error 37 us; skew 500 ppm
Aug 06 04:39:53 dist-test-slave-vkjp krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991193, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:53.260684 6703 init.cc:377] Logged in from keytab as oryx/127.31.166.130@KRBTEST.COM (short username oryx)
I20260806 04:39:53.261546 6703 webserver.cc:460] Webserver started at http://127.31.166.130:39977/ using document root <none> and password file <none>
I20260806 04:39:53.262110 6703 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:53.262202 6703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:53.262437 6703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:53.264585 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "ee19bfd1525e40929731e010edb80f54"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "696f65b06e559dd5a58c141c77d2490d"
server_key_iv: "67f7ab38fb9af80ac0c09e2298e5c821"
server_key_version: "encryptionkey@0"
I20260806 04:39:53.265239 6703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "ee19bfd1525e40929731e010edb80f54"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "696f65b06e559dd5a58c141c77d2490d"
server_key_iv: "67f7ab38fb9af80ac0c09e2298e5c821"
server_key_version: "encryptionkey@0"
I20260806 04:39:53.269078 6703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:39:53.271517 6719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:53.272776 6703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:39:53.272957 6703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "ee19bfd1525e40929731e010edb80f54"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "696f65b06e559dd5a58c141c77d2490d"
server_key_iv: "67f7ab38fb9af80ac0c09e2298e5c821"
server_key_version: "encryptionkey@0"
I20260806 04:39:53.273077 6703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:53.304034 6703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:53.307578 6703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:53.307816 6703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:53.308457 6703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:53.309335 6703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:53.309435 6703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:53.309513 6703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:53.309566 6703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:53.318816 6703 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:35665
I20260806 04:39:53.318849 6840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:35665 every 8 connection(s)
I20260806 04:39:53.319857 6703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:39:53.326018 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 6703
I20260806 04:39:53.326143 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:39:53.326448 32410 external_mini_cluster.cc:1428] Setting key 43454f9a447fb7ff8fa63e365df86327
Aug 06 04:39:53 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991193, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
I20260806 04:39:53.335407 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.321861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:51773 (local address 127.31.166.190:43117)
0806 04:39:53.322045 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:53.322051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:53.322917 (+ 866us) server_negotiation.cc:407] Connection header received
0806 04:39:53.324338 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:53.324341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:53.324393 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:53.324478 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:53.325786 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.326641 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.328046 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.328382 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.328508 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.331371 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:53.331378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:53.331385 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:53.331424 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:53.333191 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.333786 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.333789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.333791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.333876 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.334227 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.334230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.334232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.334434 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:53.334545 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:53.335062 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 04:39:53.335226 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 04:39:53.336326 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.322266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.130:51773)
0806 04:39:53.322746 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:53.322792 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:53.324103 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:53.324671 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:53.324678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:53.325057 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:53.325595 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.325609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.326777 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.326779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.327854 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.327865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.328540 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.328543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.328686 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.329335 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:53.329351 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:53.331070 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:53.333372 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.333377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.333390 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.333669 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.333973 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.333976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.333978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.334119 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.334543 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:53.334550 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:53.334806 (+ 256us) client_negotiation.cc:769] Sending connection context
0806 04:39:53.335003 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 04:39:53.335369 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for oryx/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:39:53.337603 6842 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43117
I20260806 04:39:53.337867 6842 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:53.338404 6842 heartbeater.cc:511] Master 127.31.166.190:43117 requested a full tablet report, sending...
I20260806 04:39:53.339586 6488 ts_manager.cc:194] Registered new tserver with Master: ee19bfd1525e40929731e010edb80f54 (127.31.166.130:35665)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.166.131@KRBTEST.COM" created.
I20260806 04:39:53.340446 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:51773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.131.keytab.
Entry for principal oryx/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.131.keytab.
I20260806 04:39:53.355895 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43117
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.131.keytab
--principal=oryx/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:53.465971 6847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:53.466265 6847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:53.466356 6847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:53.470393 6847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:53.470482 6847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:53.470573 6847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:39:53.475788 6847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.131.keytab
--principal=oryx/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:53.476949 6847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:53.478053 6847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:53.487954 6856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:53.487994 6854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:53.487954 6853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:53.488219 6847 server_base.cc:1034] running on GCE node
I20260806 04:39:53.488670 6847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:53.489329 6847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:53.490485 6847 hybrid_clock.cc:648] HybridClock initialized: now 1785991193490473 us; error 24 us; skew 500 ppm
Aug 06 04:39:53 dist-test-slave-vkjp krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991193, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:53.493239 6847 init.cc:377] Logged in from keytab as oryx/127.31.166.131@KRBTEST.COM (short username oryx)
I20260806 04:39:53.494117 6847 webserver.cc:460] Webserver started at http://127.31.166.131:39259/ using document root <none> and password file <none>
I20260806 04:39:53.494626 6847 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:53.494684 6847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:53.494854 6847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:53.496898 6847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "595843906abc4b7386232904fb4280de"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "4d31df220cbac47ca3f58fc65b04f8d9"
server_key_iv: "159fccb22af25a486811b56d8949a3ed"
server_key_version: "encryptionkey@0"
I20260806 04:39:53.497419 6847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "595843906abc4b7386232904fb4280de"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "4d31df220cbac47ca3f58fc65b04f8d9"
server_key_iv: "159fccb22af25a486811b56d8949a3ed"
server_key_version: "encryptionkey@0"
I20260806 04:39:53.501138 6847 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:39:53.503736 6864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:53.505103 6847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:39:53.505239 6847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "595843906abc4b7386232904fb4280de"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "4d31df220cbac47ca3f58fc65b04f8d9"
server_key_iv: "159fccb22af25a486811b56d8949a3ed"
server_key_version: "encryptionkey@0"
I20260806 04:39:53.505357 6847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:53.525346 6847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:53.528373 6847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:53.528580 6847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:53.529182 6847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:53.530022 6847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:53.530100 6847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:53.530241 6847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:53.530299 6847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:53.539462 6847 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:37535
I20260806 04:39:53.539482 6990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:37535 every 8 connection(s)
I20260806 04:39:53.540352 6847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:39:53.542555 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 6847
I20260806 04:39:53.542680 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:39:53.542994 32410 external_mini_cluster.cc:1428] Setting key 671bf5082690ee5689dfa5ec712ed2f3
Aug 06 04:39:53 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991193, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
I20260806 04:39:53.555483 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.542357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:55287 (local address 127.31.166.190:43117)
0806 04:39:53.542563 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:53.542570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:53.543536 (+ 966us) server_negotiation.cc:407] Connection header received
0806 04:39:53.545114 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:53.545118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:53.545179 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:53.545296 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:53.546561 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.547225 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.548488 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.548734 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.548858 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.551708 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:53.551712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:53.551715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:53.551746 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:53.553338 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.553879 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.553882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.553884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.553968 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.554321 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.554326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.554327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.554525 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:53.554646 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:53.555156 (+ 510us) server_negotiation.cc:299] Negotiation successful
0806 04:39:53.555305 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 04:39:53.556469 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.542799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.131:55287)
0806 04:39:53.543373 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:53.543430 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:53.544891 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:53.545491 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:53.545499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:53.545889 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:53.546404 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.546418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.547373 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.547377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.548349 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.548359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.548912 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.548916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.549095 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.549685 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:53.549702 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:53.551547 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:53.553468 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.553474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.553498 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.553764 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.554069 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.554073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.554075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.554222 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.554633 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:53.554640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:53.554895 (+ 255us) client_negotiation.cc:769] Sending connection context
0806 04:39:53.555090 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 04:39:53.555442 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":150,"threads_started":1}
I20260806 04:39:53.557642 6991 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43117
I20260806 04:39:53.557897 6991 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:53.558358 6991 heartbeater.cc:511] Master 127.31.166.190:43117 requested a full tablet report, sending...
I20260806 04:39:53.559432 6488 ts_manager.cc:194] Registered new tserver with Master: 595843906abc4b7386232904fb4280de (127.31.166.131:37535)
I20260806 04:39:53.560043 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:55287
I20260806 04:39:53.569027 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.166.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.166.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.166.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.189.keytab.
Entry for principal oryx/127.31.166.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.189.keytab.
I20260806 04:39:53.598682 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.189:39575
--webserver_interface=127.31.166.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.189.keytab
--principal=oryx/127.31.166.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:43117,127.31.166.189:39575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:39:53.722558 6996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:53.722898 6996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:53.722990 6996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:53.727120 6996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:53.727232 6996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:53.727285 6996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:53.727348 6996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:53.727391 6996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:39:53.733003 6996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:43117,127.31.166.189:39575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.189.keytab
--principal=oryx/127.31.166.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.189:39575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.166.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
W20260806 04:39:53.733292 6996 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.166.190:43117,127.31.166.189:39575'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 04:39:53.734261 6996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:53.735383 6996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:39:53.744606 6996 server_base.cc:1034] running on GCE node
W20260806 04:39:53.744582 7002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:53.744694 7001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:53.744830 7004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:53.745543 6996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:53.746836 6996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:53.748047 6996 hybrid_clock.cc:648] HybridClock initialized: now 1785991193748009 us; error 50 us; skew 500 ppm
Aug 06 04:39:53 dist-test-slave-vkjp krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991193, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:53.751300 6996 init.cc:377] Logged in from keytab as oryx/127.31.166.189@KRBTEST.COM (short username oryx)
I20260806 04:39:53.752298 6996 webserver.cc:460] Webserver started at http://127.31.166.189:44961/ using document root <none> and password file <none>
I20260806 04:39:53.752911 6996 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:53.753008 6996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:53.753247 6996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:53.755448 6996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/data/instance:
uuid: "37c687289f6c45a785cb45ca13c8f276"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "d4decd86f2cde3d072a1d93ab1a91af9"
server_key_iv: "531a1a6a05b69ef7f1c2c0fbea18f0be"
server_key_version: "encryptionkey@0"
I20260806 04:39:53.756042 6996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/wal/instance:
uuid: "37c687289f6c45a785cb45ca13c8f276"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "d4decd86f2cde3d072a1d93ab1a91af9"
server_key_iv: "531a1a6a05b69ef7f1c2c0fbea18f0be"
server_key_version: "encryptionkey@0"
I20260806 04:39:53.760133 6996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:39:53.762564 7014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:53.763886 6996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:39:53.764128 6996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/wal
uuid: "37c687289f6c45a785cb45ca13c8f276"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "d4decd86f2cde3d072a1d93ab1a91af9"
server_key_iv: "531a1a6a05b69ef7f1c2c0fbea18f0be"
server_key_version: "encryptionkey@0"
I20260806 04:39:53.764249 6996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:53.782354 6996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:53.785933 6996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:53.786203 6996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:53.793776 6996 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.189:39575
I20260806 04:39:53.794056 7074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.189:39575 every 8 connection(s)
I20260806 04:39:53.794938 6996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/data/info.pb
I20260806 04:39:53.795864 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 6996
I20260806 04:39:53.798599 7075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:53.802712 7075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:39:53 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.166.189@KRBTEST.COM
Aug 06 04:39:53 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991193, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.189@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
I20260806 04:39:53.811389 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.797517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60590 (local address 127.31.166.189:39575)
0806 04:39:53.798003 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:53.798020 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:53.798056 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:39:53.798765 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:53.798782 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:53.799176 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:53.799507 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:53.800317 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.801303 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.802193 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.802655 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.802836 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.805591 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:53.805598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:53.805613 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:53.805646 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:53.807892 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.808479 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.808485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.808497 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.808681 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.809047 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.809051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.809052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.809594 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:53.809783 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:53.810123 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 04:39:53.810417 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":163,"threads_started":1}
I20260806 04:39:53.816321 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.804516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.31.166.190:43117)
0806 04:39:53.804676 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:53.804681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:53.805373 (+ 692us) server_negotiation.cc:407] Connection header received
0806 04:39:53.805547 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:53.805552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:53.805611 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:53.805655 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:53.806502 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.807201 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.808135 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.808541 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.808648 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.811228 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:53.811232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:53.811235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:53.811264 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:53.814056 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.814504 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.814508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.814510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.814602 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.814983 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.814987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.814989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.815309 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:53.815424 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:53.816012 (+ 588us) server_negotiation.cc:299] Negotiation successful
0806 04:39:53.816165 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:39:53.816370 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.804458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50186)
0806 04:39:53.805228 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:53.805281 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:53.805410 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:53.805788 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:53.805792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:53.806246 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:53.806375 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.806381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.807328 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.807331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.808010 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.808020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.808659 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.808663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.808782 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.809384 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:53.809398 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:53.811070 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:53.814172 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.814179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.814183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.814403 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.814696 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.814698 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.814699 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.814775 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.815454 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:53.815464 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:53.815700 (+ 236us) client_negotiation.cc:769] Sending connection context
0806 04:39:53.815963 (+ 263us) client_negotiation.cc:240] Negotiation successful
0806 04:39:53.816214 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":106,"threads_started":1}
I20260806 04:39:53.817462 7075 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:39:53 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991193, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.189@KRBTEST.COM for oryx/127.31.166.189@KRBTEST.COM
I20260806 04:39:53.826586 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.817978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:39575 (local address 127.0.0.1:60594)
0806 04:39:53.818134 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:53.818147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:53.818212 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:53.818411 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:53.818414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:53.818595 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:53.818707 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.818720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.820102 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.820106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.820956 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.820967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.821524 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.821528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.821641 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.822114 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:53.822126 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:53.823421 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:53.825077 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.825079 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.825080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.825295 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.825612 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.825615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.825616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.825714 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.826118 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:53.826121 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:53.826201 (+ 80us) client_negotiation.cc:769] Sending connection context
0806 04:39:53.826299 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 04:39:53.826457 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 04:39:53.826664 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.818068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.31.166.189:39575)
0806 04:39:53.818183 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:53.818186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:53.818197 (+ 11us) server_negotiation.cc:407] Connection header received
0806 04:39:53.818288 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:53.818290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:53.818323 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:53.818391 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:53.818913 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.819967 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.821136 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.821410 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.821492 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.823556 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:53.823559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:53.823561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:53.823581 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:53.824955 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.825419 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.825421 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.825422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.825492 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.825811 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.825813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.825815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.825996 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:53.826074 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:53.826389 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 04:39:53.826528 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":41}
I20260806 04:39:53.829578 7075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: Bootstrap starting.
I20260806 04:39:53.831959 7075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:53.832671 7075 log.cc:824] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:53.834584 7075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: No bootstrap required, opened a new log
I20260806 04:39:53.837543 7075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } }
I20260806 04:39:53.837757 7075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:53.837837 7075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37c687289f6c45a785cb45ca13c8f276, State: Initialized, Role: FOLLOWER
I20260806 04:39:53.838207 7075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } }
I20260806 04:39:53.838662 7081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } } }
I20260806 04:39:53.838786 7081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:39:53.839365 7075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 04:39:53.844061 7093 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:39:53.844148 7093 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:39:53.846043 6996 master_runner.cc:428] Detected that this master 37c687289f6c45a785cb45ca13c8f276 is joining an existing cluster
I20260806 04:39:53.846141 6996 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.166.189:39575
I20260806 04:39:53.858135 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.848420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:39575 (local address 127.0.0.1:60604)
0806 04:39:53.848737 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:53.848751 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:53.848849 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:53.849836 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:53.849839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:53.850058 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:53.850197 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.850204 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.851261 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.851265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.852205 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.852215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.852938 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.852942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.853052 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.853576 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:53.853598 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:53.854249 (+ 651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:53.856391 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.856396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.856399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.856655 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.856979 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.856983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.856985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.857072 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.857493 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:53.857496 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:53.857665 (+ 169us) client_negotiation.cc:769] Sending connection context
0806 04:39:53.857771 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 04:39:53.857902 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":25,"spinlock_wait_cycles":47744,"thread_start_us":131,"threads_started":1}
I20260806 04:39:53.858523 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.848828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.31.166.190:43117)
0806 04:39:53.849097 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:53.849101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:53.849113 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:39:53.849148 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:53.849150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:53.849188 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:53.849291 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:53.849800 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.850572 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.851842 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.852176 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.852286 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.854732 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:53.854736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:53.854739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:53.854764 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:53.856498 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.856984 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.856988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.856990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.857077 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.857421 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.857428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.857433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.857740 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:53.857874 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:53.858217 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 04:39:53.858372 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260806 04:39:53.858804 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.848978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.31.166.189:39575)
0806 04:39:53.849094 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:53.849099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:53.849112 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:39:53.849143 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:53.849146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:53.849185 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:53.849290 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:53.850345 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.851116 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.852351 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.852709 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.852777 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.854405 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:53.854409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:53.854412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:53.854436 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:53.856251 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.856773 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.856779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.856781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.856863 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.857167 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.857173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.857175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.857398 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:53.857506 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:53.857878 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 04:39:53.858005 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:39:53.859066 7099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.848337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50196)
0806 04:39:53.848669 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:53.848686 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:53.848790 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:53.849323 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:53.849326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:53.849550 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:53.849662 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.849668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.850695 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.850699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.851712 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.851723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.852321 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.852328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.852425 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.853576 (+ 1151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:53.853597 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:53.854243 (+ 646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:53.856611 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.856617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.856620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.856885 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.857187 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.857190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.857192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.857309 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.857919 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:53.857922 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:53.858012 (+ 90us) client_negotiation.cc:769] Sending connection context
0806 04:39:53.858721 (+ 709us) client_negotiation.cc:240] Negotiation successful
0806 04:39:53.858869 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":47,"spinlock_wait_cycles":8192,"thread_start_us":101,"threads_started":1}
Aug 06 04:39:53 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991192, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.190@KRBTEST.COM for oryx/127.31.166.189@KRBTEST.COM
WARNING: no policy specified for oryx/127.31.166.188@KRBTEST.COM; defaulting to no policy
I20260806 04:39:53.875730 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.864839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.31.166.189:39575)
0806 04:39:53.864956 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:53.864960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:53.864972 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:39:53.865027 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:53.865029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:53.865069 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:53.865160 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:53.865927 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.866416 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.867462 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.867685 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.867796 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.870697 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:53.870703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:53.870709 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:53.870743 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:53.873674 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.874268 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.874272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.874274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.874373 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.874688 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.874693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.874695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.874938 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:53.875022 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:53.875462 (+ 440us) server_negotiation.cc:299] Negotiation successful
0806 04:39:53.875602 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:39:53.876029 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.864056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:39575 (local address 127.0.0.1:60616)
0806 04:39:53.864492 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:53.864544 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:53.864666 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:53.865200 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:53.865206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:53.865638 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:53.865792 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.865801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.866534 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.866537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.867348 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.867358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.867800 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.867804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.867928 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.868537 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:53.868557 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:53.870534 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:53.873799 (+ 3265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.873810 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.873817 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.874146 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.874479 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.874482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.874484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.874585 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.875046 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:53.875053 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:53.875225 (+ 172us) client_negotiation.cc:769] Sending connection context
0806 04:39:53.875398 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 04:39:53.875843 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.166.188@KRBTEST.COM" created.
I20260806 04:39:53.877929 6488 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.166.189:39575: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "37c687289f6c45a785cb45ca13c8f276"
member_type: NON_VOTER
last_known_addr {
host: "127.31.166.189"
port: 39575
}
attrs {
promote: true
}
}
dest_uuid: "824064c26018499f879fdf4970b0b8e4"
cas_config_opid_index: -1
I20260806 04:39:53.878746 6488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: NON_VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: true } }
I20260806 04:39:53.881135 7105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 37c687289f6c45a785cb45ca13c8f276 (127.31.166.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: NON_VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: true } } }
I20260806 04:39:53.881922 7044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:53.882985 7105 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.166.189:39575
I20260806 04:39:53.883741 6996 master.cc:537] Master@127.31.166.189:39575 shutting down...
I20260806 04:39:53.883800 7044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 FOLLOWER]: Refusing update from remote peer 824064c26018499f879fdf4970b0b8e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 04:39:53.884500 7105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: NON_VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: true } } }
I20260806 04:39:53.884611 7105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:53.884778 7105 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: NON_VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 04:39:53.886756 6472 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 04:39:53.887779 6472 consensus_peers.cc:487] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 -> Peer 37c687289f6c45a785cb45ca13c8f276 (127.31.166.189:39575): Couldn't send request to peer 37c687289f6c45a785cb45ca13c8f276. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:39:53.889405 6996 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:53.889739 6996 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:53.889881 6996 tablet_replica.cc:330] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.166.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.188.keytab.
Entry for principal oryx/127.31.166.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.188.keytab.
I20260806 04:39:53.898590 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.188:44279
--webserver_interface=127.31.166.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.188.keytab
--principal=oryx/127.31.166.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:43117,127.31.166.189:39575,127.31.166.188:44279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:39:53.906355 6996 master.cc:552] Master@127.31.166.189:39575 shutdown complete.
I20260806 04:39:53.906438 6996 master_runner.cc:305] Clearing existing system tablet
I20260806 04:39:53.906775 6996 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:53.906855 6996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:53.908702 6996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 04:39:53.909971 7111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:53.910288 6996 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:53.910468 6996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 04:39:53.910521 6996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/wal
uuid: "37c687289f6c45a785cb45ca13c8f276"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "d4decd86f2cde3d072a1d93ab1a91af9"
server_key_iv: "531a1a6a05b69ef7f1c2c0fbea18f0be"
server_key_version: "encryptionkey@0"
I20260806 04:39:53.913168 6996 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:53.915793 6996 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 04:39:53.915897 6996 log.cc:1196] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 04:39:53.916183 6996 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: Deleting consensus metadata
I20260806 04:39:53.916574 6996 master_runner.cc:315] Copying system tablet from 127.31.166.190:43117
I20260806 04:39:53.917917 6996 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: tablet copy: Beginning tablet copy session from remote peer at address 127.31.166.190:43117
I20260806 04:39:53.928092 7117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.918545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50200)
0806 04:39:53.919286 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:53.919302 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:53.919436 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:53.919750 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:53.919753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:53.919976 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:53.920120 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.920129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.921266 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.921269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.922190 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:53.922203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.922872 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.922876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:53.922977 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.923627 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:53.923644 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:53.924074 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:53.926293 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.926297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.926299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.926569 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.926895 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:53.926898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:53.926900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:53.926993 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:53.927505 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:53.927508 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:53.927613 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 04:39:53.927728 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 04:39:53.927872 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":102,"threads_started":1}
I20260806 04:39:53.928104 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:53.918685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.31.166.190:43117)
0806 04:39:53.918867 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:53.918875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:53.919403 (+ 528us) server_negotiation.cc:407] Connection header received
0806 04:39:53.919532 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:53.919536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:53.919598 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:53.919720 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:53.920255 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.921132 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.922329 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:53.922716 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:53.922856 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:53.924253 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:53.924257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:53.924262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:53.924296 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:53.926166 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.926678 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.926682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.926685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.926781 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:53.927146 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:53.927150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:53.927151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:53.927394 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:53.927504 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:53.927804 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 04:39:53.927942 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 04:39:53.929239 6511 tablet_copy_service.cc:140] P 824064c26018499f879fdf4970b0b8e4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 37c687289f6c45a785cb45ca13c8f276 ({username='oryx', principal='oryx/127.31.166.189@KRBTEST.COM'} at 127.0.0.1:50200)
I20260806 04:39:53.929428 6511 tablet_copy_service.cc:161] P 824064c26018499f879fdf4970b0b8e4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 37c687289f6c45a785cb45ca13c8f276 at {username='oryx', principal='oryx/127.31.166.189@KRBTEST.COM'} at 127.0.0.1:50200: session id = 37c687289f6c45a785cb45ca13c8f276-00000000000000000000000000000000
I20260806 04:39:53.931291 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:39:53.933002 6996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 04:39:53.935523 6996 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: tablet copy: Starting download of 0 data blocks...
I20260806 04:39:53.935802 6996 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: tablet copy: Starting download of 1 WAL segments...
I20260806 04:39:53.937899 6996 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:39:53.940030 6511 tablet_copy_service.cc:342] P 824064c26018499f879fdf4970b0b8e4: Request end of tablet copy session 37c687289f6c45a785cb45ca13c8f276-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.166.189@KRBTEST.COM'} at 127.0.0.1:50200
I20260806 04:39:53.940202 6511 tablet_copy_service.cc:433] P 824064c26018499f879fdf4970b0b8e4: ending tablet copy session 37c687289f6c45a785cb45ca13c8f276-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 37c687289f6c45a785cb45ca13c8f276
I20260806 04:39:53.947916 6996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:53.949772 7118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:53.949800 7121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:53.951399 7119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:53.951764 6996 server_base.cc:1034] running on GCE node
I20260806 04:39:53.951925 6996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:53.952224 6996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:53.953380 6996 hybrid_clock.cc:648] HybridClock initialized: now 1785991193953356 us; error 44 us; skew 500 ppm
Aug 06 04:39:53 dist-test-slave-vkjp krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991193, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:53.955157 6996 init.cc:377] Logged in from keytab as oryx/127.31.166.189@KRBTEST.COM (short username oryx)
I20260806 04:39:53.955649 6996 webserver.cc:460] Webserver started at http://127.31.166.189:34317/ using document root <none> and password file <none>
I20260806 04:39:53.955835 6996 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:53.955870 6996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:53.957482 6996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260806 04:39:53.959586 7129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:53.959878 6996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:39:53.959952 6996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/wal
uuid: "37c687289f6c45a785cb45ca13c8f276"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "d4decd86f2cde3d072a1d93ab1a91af9"
server_key_iv: "531a1a6a05b69ef7f1c2c0fbea18f0be"
server_key_version: "encryptionkey@0"
I20260806 04:39:53.960026 6996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:53.992659 6996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:53.995450 6996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:54.001587 6996 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.189:39575
I20260806 04:39:54.001654 7189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.189:39575 every 8 connection(s)
I20260806 04:39:54.002516 6996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-1/data/info.pb
I20260806 04:39:54.004876 7190 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:39:54.006047 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: Bootstrap starting.
I20260806 04:39:54.018232 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:39:54.018617 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: Bootstrap complete.
I20260806 04:39:54.019358 7190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: NON_VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: true } }
I20260806 04:39:54.019524 7190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 37c687289f6c45a785cb45ca13c8f276, State: Initialized, Role: LEARNER
I20260806 04:39:54.019670 7190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: NON_VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: true } }
I20260806 04:39:54.020268 7190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:54.020604 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: NON_VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: true } } }
I20260806 04:39:54.020866 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: This master's current role is: LEARNER
I20260806 04:39:54.023658 7204 catalog_manager.cc:1259] Loaded cluster ID: b20d78810fde47c7ae5a4e7969b72dc4
I20260806 04:39:54.023741 7204 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: loading cluster ID for follower catalog manager: success
I20260806 04:39:54.025547 7204 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: acquiring CA information for follower catalog manager: success
I20260806 04:39:54.026752 7204 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 04:39:54.047662 7109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:54.047919 7109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:54.047987 7109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:54.052074 7109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:39:54.052126 7109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:54.052145 7109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:54.052162 7109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:39:54.052206 7109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:39:54.057540 7109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:43117,127.31.166.189:39575,127.31.166.188:44279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.188.keytab
--principal=oryx/127.31.166.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.188:44279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.166.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:54.058660 7109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:54.059723 7109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:54.068019 7209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:54.068028 7210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:54.068176 7109 server_base.cc:1034] running on GCE node
W20260806 04:39:54.068367 7212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:54.068939 7109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:54.069947 7109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:54.071165 7109 hybrid_clock.cc:648] HybridClock initialized: now 1785991194071145 us; error 42 us; skew 500 ppm
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:54.073927 7109 init.cc:377] Logged in from keytab as oryx/127.31.166.188@KRBTEST.COM (short username oryx)
I20260806 04:39:54.075017 7109 webserver.cc:460] Webserver started at http://127.31.166.188:42153/ using document root <none> and password file <none>
I20260806 04:39:54.075592 7109 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:54.075654 7109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:54.075817 7109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:39:54.077920 7109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/data/instance:
uuid: "ee8e60bd63314a43b4fea6f479518272"
format_stamp: "Formatted at 2026-08-06 04:39:54 on dist-test-slave-vkjp"
server_key: "cc9a0eb11b848e5a6027ee95a8877ee3"
server_key_iv: "8829285387a9694b9a28104ed71651c4"
server_key_version: "encryptionkey@0"
I20260806 04:39:54.078466 7109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/wal/instance:
uuid: "ee8e60bd63314a43b4fea6f479518272"
format_stamp: "Formatted at 2026-08-06 04:39:54 on dist-test-slave-vkjp"
server_key: "cc9a0eb11b848e5a6027ee95a8877ee3"
server_key_iv: "8829285387a9694b9a28104ed71651c4"
server_key_version: "encryptionkey@0"
I20260806 04:39:54.082350 7109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:39:54.085242 7219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:54.086612 7109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:39:54.086755 7109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/wal
uuid: "ee8e60bd63314a43b4fea6f479518272"
format_stamp: "Formatted at 2026-08-06 04:39:54 on dist-test-slave-vkjp"
server_key: "cc9a0eb11b848e5a6027ee95a8877ee3"
server_key_iv: "8829285387a9694b9a28104ed71651c4"
server_key_version: "encryptionkey@0"
I20260806 04:39:54.086872 7109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:54.108317 7109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:54.110265 6698 heartbeater.cc:503] Master 127.31.166.190:43117 was elected leader, sending a full tablet report...
I20260806 04:39:54.111825 7109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:54.112037 7109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:54.119300 7109 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.188:44279
I20260806 04:39:54.119642 7274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.188:44279 every 8 connection(s)
I20260806 04:39:54.120308 7109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/data/info.pb
I20260806 04:39:54.123414 7275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:54.125980 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 7109
I20260806 04:39:54.127578 7275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.188@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.166.188@KRBTEST.COM
I20260806 04:39:54.140974 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.129694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.31.166.190:43117)
0806 04:39:54.129835 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.129842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.129960 (+ 118us) server_negotiation.cc:407] Connection header received
0806 04:39:54.130103 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.130108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.130172 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.130284 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.131522 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.132123 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.133353 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.133566 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.133652 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.136393 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.136397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.136401 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.136435 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.138390 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.139139 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.139143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.139145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.139256 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.139620 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.139625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.139627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.139839 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.139977 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.140607 (+ 630us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.140772 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:39:54.142056 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.127602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49258 (local address 127.31.166.188:44279)
0806 04:39:54.128049 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.128071 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.128117 (+ 46us) server_negotiation.cc:407] Connection header received
0806 04:39:54.129003 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.129024 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.129414 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.130657 (+ 1243us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.131128 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.132348 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.133308 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.133716 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.133890 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.136828 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.136834 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.136860 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.136893 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.139131 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.139676 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.139686 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.139691 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.139804 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.140135 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.140138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.140140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.140435 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.140868 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.140985 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.141136 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":288,"thread_start_us":113,"threads_started":1}
I20260806 04:39:54.142056 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.129456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50216)
0806 04:39:54.129817 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.129870 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.129993 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.130321 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.130327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.130753 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.131321 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.131335 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.132261 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.132266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.133160 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.133170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.133712 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.133716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.133890 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.134625 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.134641 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.136194 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.138538 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.138546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.138556 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.138964 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.139374 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.139376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.139377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.139510 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.139971 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.139977 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.140296 (+ 319us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.140535 (+ 239us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.140938 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":73,"spinlock_wait_cycles":12288,"thread_start_us":130,"threads_started":1}
I20260806 04:39:54.143733 7275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.188@KRBTEST.COM for oryx/127.31.166.189@KRBTEST.COM
I20260806 04:39:54.152673 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.144270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:39575 (local address 127.0.0.1:60626)
0806 04:39:54.144402 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.144418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.144502 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.144922 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.144924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.145060 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.145162 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.145168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.146209 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.146212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.146780 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.146790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.147321 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.147324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.147411 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.147879 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.147891 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.149202 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.151025 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.151027 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.151029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.151317 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.151638 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.151641 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.151643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.151728 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.152116 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.152119 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.152249 (+ 130us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.152418 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.152558 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 04:39:54.152772 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.144405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.31.166.189:39575)
0806 04:39:54.144722 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.144727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.144739 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:39:54.144783 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.144786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.144827 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.144903 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.145368 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.146088 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.146904 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.147207 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.147334 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.149338 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.149341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.149343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.149390 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.150923 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.151436 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.151439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.151441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.151533 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.151827 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.151831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.151832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.152015 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.152124 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.152504 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.152606 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":103,"threads_started":1}
I20260806 04:39:54.153537 7275 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.188@KRBTEST.COM for oryx/127.31.166.188@KRBTEST.COM
I20260806 04:39:54.161402 7278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.154047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:44279 (local address 127.0.0.1:49270)
0806 04:39:54.154183 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.154200 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.154281 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.154582 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.154584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.154723 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.154813 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.154818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.155585 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.155588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.156186 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.156194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.156616 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.156619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.156681 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.157096 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.157108 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.158301 (+ 1193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.159963 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.159966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.159967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.160150 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.160468 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.160469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.160471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.160585 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.161022 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.161025 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.161103 (+ 78us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.161197 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.161291 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 04:39:54.161584 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.154243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49270 (local address 127.31.166.188:44279)
0806 04:39:54.154353 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.154357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.154369 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:39:54.154434 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.154436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.154469 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.154537 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.154918 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.155475 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.156291 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.156518 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.156597 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.158451 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.158454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.158456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.158478 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.159866 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.160264 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.160267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.160269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.160381 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.160668 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.160691 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.160697 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.160888 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.160971 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.161273 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.161437 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:39:54.164820 7275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: Bootstrap starting.
I20260806 04:39:54.167057 7275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:54.167809 7275 log.cc:824] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:54.169718 7275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: No bootstrap required, opened a new log
I20260806 04:39:54.172674 7275 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } }
I20260806 04:39:54.172874 7275 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:54.172909 7275 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee8e60bd63314a43b4fea6f479518272, State: Initialized, Role: FOLLOWER
I20260806 04:39:54.173255 7275 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } }
I20260806 04:39:54.173681 7281 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } } }
I20260806 04:39:54.173818 7281 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:39:54.174492 7275 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 04:39:54.179549 7292 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:39:54.179641 7292 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:39:54.182971 7109 master_runner.cc:428] Detected that this master ee8e60bd63314a43b4fea6f479518272 is joining an existing cluster
I20260806 04:39:54.183048 7109 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.166.188:44279
I20260806 04:39:54.195053 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.186259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.31.166.188:44279)
0806 04:39:54.186358 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.186363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.186934 (+ 571us) server_negotiation.cc:407] Connection header received
0806 04:39:54.187110 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.187114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.187156 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.187211 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.187814 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.188597 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.190024 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.190353 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.190428 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.191841 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.191844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.191847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.191870 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.193395 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.193867 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.193870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.193871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.193949 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.194236 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.194239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.194241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.194424 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.194534 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.194819 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.194912 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 04:39:54.195065 7304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.186174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:44279 (local address 127.0.0.1:49274)
0806 04:39:54.186865 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.186878 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.187000 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.187355 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.187361 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.187560 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.187709 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.187719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.188727 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.188730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.189864 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.189875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.190592 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.190598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.190692 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.191321 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.191335 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.191693 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.193524 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.193527 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.193529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.193728 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.194043 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.194048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.194050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.194136 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.194522 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.194525 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.194607 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.194719 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.194860 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":138,"thread_start_us":74,"threads_started":1}
I20260806 04:39:54.196182 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 6558
I20260806 04:39:54.209380 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.185773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.31.166.190:43117)
0806 04:39:54.185936 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.185941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.186615 (+ 674us) server_negotiation.cc:407] Connection header received
0806 04:39:54.186732 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.186736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.186805 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.186905 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.187904 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.188846 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.190220 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.190626 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.190712 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.205381 (+ 14669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.205386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.205389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.205427 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.207170 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.207734 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.207738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.207740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.207837 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.208203 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.208206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.208208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.208435 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.208510 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.209066 (+ 556us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.209202 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 04:39:54.209462 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.185831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.31.166.189:39575)
0806 04:39:54.185961 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.185964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.186587 (+ 623us) server_negotiation.cc:407] Connection header received
0806 04:39:54.186743 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.186747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.186799 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.186867 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.187801 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.188597 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.190007 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.190358 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.190428 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.204671 (+ 14243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.204675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.204678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.204712 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.207021 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.207735 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.207738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.207740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.207829 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.208190 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.208194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.208195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.208435 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.208510 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.209074 (+ 564us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.209202 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 04:39:54.210050 7303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.185727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:39575 (local address 127.0.0.1:60642)
0806 04:39:54.186500 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.186515 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.186636 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.187031 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.187036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.187441 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.187648 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.187658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.188727 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.188730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.189865 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.189875 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.190656 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.190660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.190736 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.204006 (+ 13270us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.204028 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.204455 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.207198 (+ 2743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.207204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.207210 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.207608 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.207953 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.207956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.207958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.208075 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.208636 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.208639 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.208857 (+ 218us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.209598 (+ 741us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.209865 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":195,"spinlock_wait_cycles":37760,"thread_start_us":116,"threads_started":1}
I20260806 04:39:54.210341 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:39873
--local_ip_for_outbound_sockets=127.31.166.129
--tserver_master_addrs=127.31.166.190:43117
--webserver_port=46163
--webserver_interface=127.31.166.129
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.129.keytab
--principal=oryx/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.166.190:43117,127.31.166.189:39575
--tserver_master_addrs=127.31.166.190:43117,127.31.166.189:39575,127.31.166.188:44279 with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:39:54.210745 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.186070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50230)
0806 04:39:54.186500 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.186515 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.186625 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.187043 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.187047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.187490 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.187730 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.187740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.189005 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.189009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.190065 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.190077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.191003 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.191007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.191148 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.204677 (+ 13529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.204692 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.205159 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.207285 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.207291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.207293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.207608 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.207953 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.207956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.207958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.208082 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.208636 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.208639 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.208857 (+ 218us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.210061 (+ 1204us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.210563 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":86,"spinlock_wait_cycles":14336,"thread_start_us":87,"threads_started":1}
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991192, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.190@KRBTEST.COM for oryx/127.31.166.188@KRBTEST.COM
I20260806 04:39:54.222613 7277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.214725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49290 (local address 127.31.166.188:44279)
0806 04:39:54.214852 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.214855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.214868 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:39:54.214974 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.214976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.215017 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.215144 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.215548 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.216037 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.216897 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.217114 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.217223 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.219325 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.219329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.219331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.219359 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.220930 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.221384 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.221389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.221391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.221471 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.221759 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.221764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.221766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.221941 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.222062 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.222359 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.222465 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 04:39:54.222613 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.214591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:44279 (local address 127.0.0.1:49290)
0806 04:39:54.214754 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.214778 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.214864 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.215153 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.215156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.215348 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.215435 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.215440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.216147 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.216151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.216781 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.216791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.217227 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.217231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.217353 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.217790 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.217803 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.219185 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.221068 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.221073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.221075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.221279 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.221571 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.221575 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.221577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.221666 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.222091 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.222094 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.222188 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.222299 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.222453 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 04:39:54.223965 6488 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.166.188:44279: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ee8e60bd63314a43b4fea6f479518272"
member_type: NON_VOTER
last_known_addr {
host: "127.31.166.188"
port: 44279
}
attrs {
promote: true
}
}
dest_uuid: "824064c26018499f879fdf4970b0b8e4"
cas_config_opid_index: 5
I20260806 04:39:54.224552 6488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: NON_VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: true } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } }
I20260806 04:39:54.225744 7105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ee8e60bd63314a43b4fea6f479518272 (127.31.166.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: NON_VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: true } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } } }
I20260806 04:39:54.226279 7247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:54.227392 7105 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.166.188:44279
I20260806 04:39:54.227583 7105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: NON_VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: true } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } } }
I20260806 04:39:54.227685 7105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:54.228066 7247 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 1 FOLLOWER]: Refusing update from remote peer 824064c26018499f879fdf4970b0b8e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 04:39:54.228349 7109 master.cc:537] Master@127.31.166.188:44279 shutting down...
I20260806 04:39:54.229283 7106 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:54.230919 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.226067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:39575 (local address 127.0.0.1:60644)
0806 04:39:54.226535 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.226552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.226682 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.227259 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.227262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.227280 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:54.227362 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.227368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.228406 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.228409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.229476 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.229488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.230436 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.230441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.230540 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.230594 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.230700 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.230776 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387}
I20260806 04:39:54.231400 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.226309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.31.166.189:39575)
0806 04:39:54.226434 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.226439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.226610 (+ 171us) server_negotiation.cc:407] Connection header received
0806 04:39:54.226916 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.226919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.226987 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.227060 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:54.227492 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.228253 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.229588 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.230064 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.230190 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.230369 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.231161 (+ 792us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.231253 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 04:39:54.231285 6472 consensus_peers.cc:487] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 -> Peer ee8e60bd63314a43b4fea6f479518272 (127.31.166.188:44279): Couldn't send request to peer ee8e60bd63314a43b4fea6f479518272. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:39:54.232272 7160 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 LEARNER]: Refusing update from remote peer 824064c26018499f879fdf4970b0b8e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 04:39:54.232681 7106 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: NON_VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 04:39:54.234709 7160 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ee8e60bd63314a43b4fea6f479518272 (127.31.166.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: NON_VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: true } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } } }
I20260806 04:39:54.237007 7193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 824064c26018499f879fdf4970b0b8e4. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: NON_VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: true } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } } }
I20260806 04:39:54.237146 7193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: This master's current role is: LEARNER
I20260806 04:39:54.237324 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: NON_VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: true } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } } }
I20260806 04:39:54.237401 7106 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4: attempting to promote NON_VOTER 37c687289f6c45a785cb45ca13c8f276 to VOTER
I20260806 04:39:54.237434 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: This master's current role is: LEARNER
I20260806 04:39:54.237838 7106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } }
I20260806 04:39:54.239295 7159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 LEARNER]: Refusing update from remote peer 824064c26018499f879fdf4970b0b8e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 04:39:54.239712 7105 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260806 04:39:54.240120 6472 consensus_peers.cc:487] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 -> Peer ee8e60bd63314a43b4fea6f479518272 (127.31.166.188:44279): Couldn't send request to peer ee8e60bd63314a43b4fea6f479518272. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:39:54.241261 7109 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:54.241562 7109 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:54.241689 7109 tablet_replica.cc:330] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: stopping tablet replica
I20260806 04:39:54.244638 7105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 37c687289f6c45a785cb45ca13c8f276 (127.31.166.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } } }
I20260806 04:39:54.245975 7105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } } }
I20260806 04:39:54.246119 7105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:54.246270 7105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } } }
I20260806 04:39:54.246369 7105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:54.247360 7159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 37c687289f6c45a785cb45ca13c8f276 (127.31.166.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } } }
I20260806 04:39:54.248790 7307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } } }
I20260806 04:39:54.248966 7307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:39:54.261526 7109 master.cc:552] Master@127.31.166.188:44279 shutdown complete.
I20260806 04:39:54.261615 7109 master_runner.cc:305] Clearing existing system tablet
I20260806 04:39:54.261875 7109 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:54.261960 7109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:54.263290 7109 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260806 04:39:54.264303 7310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:54.264495 7109 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:54.264633 7109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 04:39:54.264680 7109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/wal
uuid: "ee8e60bd63314a43b4fea6f479518272"
format_stamp: "Formatted at 2026-08-06 04:39:54 on dist-test-slave-vkjp"
server_key: "cc9a0eb11b848e5a6027ee95a8877ee3"
server_key_iv: "8829285387a9694b9a28104ed71651c4"
server_key_version: "encryptionkey@0"
I20260806 04:39:54.266454 7109 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:54.268617 7109 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 04:39:54.268736 7109 log.cc:1196] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 04:39:54.269078 7109 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: Deleting consensus metadata
I20260806 04:39:54.269285 7109 master_runner.cc:315] Copying system tablet from 127.31.166.190:43117
I20260806 04:39:54.270537 7109 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: tablet copy: Beginning tablet copy session from remote peer at address 127.31.166.190:43117
I20260806 04:39:54.280143 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.271329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.31.166.190:43117)
0806 04:39:54.271476 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.271481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.271650 (+ 169us) server_negotiation.cc:407] Connection header received
0806 04:39:54.271848 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.271852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.271912 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.272006 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.272517 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.273153 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.274157 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.274540 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.274651 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.276172 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.276177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.276184 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.276217 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.278094 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.278656 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.278661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.278663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.278755 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.279074 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.279102 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.279104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.279417 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.279538 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.279848 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.279972 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:39:54.280155 7316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.271209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50240)
0806 04:39:54.271535 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.271551 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.271706 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.272024 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.272028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.272267 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.272385 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.272394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.273322 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.273330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.274018 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.274028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.274655 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.274659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.274777 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.275507 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.275524 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.275978 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.278215 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.278220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.278222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.278542 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.278867 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.278871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.278873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.278973 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.279526 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.279530 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.279649 (+ 119us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.279766 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.279930 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":91,"threads_started":1}
I20260806 04:39:54.281044 6511 tablet_copy_service.cc:140] P 824064c26018499f879fdf4970b0b8e4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ee8e60bd63314a43b4fea6f479518272 ({username='oryx', principal='oryx/127.31.166.188@KRBTEST.COM'} at 127.0.0.1:50240)
I20260806 04:39:54.281172 6511 tablet_copy_service.cc:161] P 824064c26018499f879fdf4970b0b8e4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ee8e60bd63314a43b4fea6f479518272 at {username='oryx', principal='oryx/127.31.166.188@KRBTEST.COM'} at 127.0.0.1:50240: session id = ee8e60bd63314a43b4fea6f479518272-00000000000000000000000000000000
I20260806 04:39:54.282088 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:39:54.283540 7109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 04:39:54.285812 7109 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: tablet copy: Starting download of 0 data blocks...
I20260806 04:39:54.286062 7109 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: tablet copy: Starting download of 1 WAL segments...
I20260806 04:39:54.287658 7109 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:39:54.289445 6511 tablet_copy_service.cc:342] P 824064c26018499f879fdf4970b0b8e4: Request end of tablet copy session ee8e60bd63314a43b4fea6f479518272-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.166.188@KRBTEST.COM'} at 127.0.0.1:50240
I20260806 04:39:54.289534 6511 tablet_copy_service.cc:433] P 824064c26018499f879fdf4970b0b8e4: ending tablet copy session ee8e60bd63314a43b4fea6f479518272-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ee8e60bd63314a43b4fea6f479518272
I20260806 04:39:54.296800 7109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:54.298605 7317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:54.298651 7321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:54.298889 7109 server_base.cc:1034] running on GCE node
W20260806 04:39:54.299405 7318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:54.299774 7109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:54.300122 7109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:54.301281 7109 hybrid_clock.cc:648] HybridClock initialized: now 1785991194301290 us; error 45 us; skew 500 ppm
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:54.303067 7109 init.cc:377] Logged in from keytab as oryx/127.31.166.188@KRBTEST.COM (short username oryx)
I20260806 04:39:54.303647 7109 webserver.cc:460] Webserver started at http://127.31.166.188:35373/ using document root <none> and password file <none>
I20260806 04:39:54.303872 7109 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:54.303936 7109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:54.305133 7109 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260806 04:39:54.306128 7328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:54.306409 7109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 04:39:54.306519 7109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/wal
uuid: "ee8e60bd63314a43b4fea6f479518272"
format_stamp: "Formatted at 2026-08-06 04:39:54 on dist-test-slave-vkjp"
server_key: "cc9a0eb11b848e5a6027ee95a8877ee3"
server_key_iv: "8829285387a9694b9a28104ed71651c4"
server_key_version: "encryptionkey@0"
I20260806 04:39:54.306635 7109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:54.312247 7109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:54.314432 7109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:54.320840 7109 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.188:44279
I20260806 04:39:54.321702 7109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/master-2/data/info.pb
I20260806 04:39:54.324625 7383 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:39:54.325559 7383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: Bootstrap starting.
I20260806 04:39:54.331476 7382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.188:44279 every 8 connection(s)
I20260806 04:39:54.336014 7383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:39:54.336401 7383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: Bootstrap complete.
I20260806 04:39:54.337073 7383 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } }
I20260806 04:39:54.337229 7383 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee8e60bd63314a43b4fea6f479518272, State: Initialized, Role: LEARNER
I20260806 04:39:54.337368 7383 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } }
I20260806 04:39:54.337787 7386 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } } }
I20260806 04:39:54.337970 7383 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:39:54.338091 7386 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [sys.catalog]: This master's current role is: LEARNER
I20260806 04:39:54.342404 7398 catalog_manager.cc:1259] Loaded cluster ID: b20d78810fde47c7ae5a4e7969b72dc4
I20260806 04:39:54.342486 7398 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: loading cluster ID for follower catalog manager: success
I20260806 04:39:54.342733 6842 heartbeater.cc:503] Master 127.31.166.190:43117 was elected leader, sending a full tablet report...
I20260806 04:39:54.345028 7398 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: acquiring CA information for follower catalog manager: success
I20260806 04:39:54.346278 7398 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 04:39:54.381556 7305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:54.381945 7305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:54.382011 7305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:54.388429 7305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:54.388528 7305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:54.388626 7305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:39:54.396816 7305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.129.keytab
--principal=oryx/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:39873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=46163
--tserver_master_addrs=127.31.166.190:43117,127.31.166.189:39575,127.31.166.188:44279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:54.398185 7305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:54.399471 7305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:54.411446 7404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:54.411324 7405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:54.411335 7407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:54.411705 7305 server_base.cc:1034] running on GCE node
I20260806 04:39:54.412200 7305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:54.412909 7305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:54.414126 7305 hybrid_clock.cc:648] HybridClock initialized: now 1785991194414104 us; error 33 us; skew 500 ppm
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:54.417435 7305 init.cc:377] Logged in from keytab as oryx/127.31.166.129@KRBTEST.COM (short username oryx)
I20260806 04:39:54.418373 7305 webserver.cc:460] Webserver started at http://127.31.166.129:46163/ using document root <none> and password file <none>
I20260806 04:39:54.418952 7305 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:54.419044 7305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:54.423789 7305 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:39:54.425987 7414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:54.427491 7305 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:39:54.427599 7305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "0c2609e68bce4b6c8edc78b221bd9f38"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "acd4a081271ee7028d74f78df48df91f"
server_key_iv: "326e615f3aaef6438739cf5ceeec3e28"
server_key_version: "encryptionkey@0"
I20260806 04:39:54.428172 7305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:54.466408 7305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:54.470160 7305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:54.470340 7305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:54.470939 7305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:54.471845 7305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:54.471897 7305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:54.471940 7305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:54.471964 7305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:54.482524 7305 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:39873
I20260806 04:39:54.482553 7537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:39873 every 8 connection(s)
I20260806 04:39:54.483695 7305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:39:54.493017 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 7305
I20260806 04:39:54.493245 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 6703
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.188@KRBTEST.COM
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.189@KRBTEST.COM
I20260806 04:39:54.502035 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.485605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:56449 (local address 127.31.166.189:39575)
0806 04:39:54.485744 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.485750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.487197 (+ 1447us) server_negotiation.cc:407] Connection header received
0806 04:39:54.488512 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.488517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.488583 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.488650 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.490180 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.490997 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.493054 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.493417 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.493542 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.497592 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.497599 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.497605 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.497641 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.499720 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.500245 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.500251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.500256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.500346 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.500685 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.500691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.500696 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.500943 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.501048 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.501646 (+ 598us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.501793 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:39:54.502354 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.485605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:45683 (local address 127.31.166.190:43117)
0806 04:39:54.485744 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.485749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.487271 (+ 1522us) server_negotiation.cc:407] Connection header received
0806 04:39:54.488512 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.488517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.488583 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.488650 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.490181 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.491005 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.492589 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.492981 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.493093 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.497610 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.497614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.497617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.497655 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.500387 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.500898 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.500902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.500904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.501021 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.501368 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.501372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.501374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.501615 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.501697 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.502074 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.502198 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:39:54.502944 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.487856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:51637 (local address 127.31.166.188:44279)
0806 04:39:54.488193 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.488196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.488233 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:39:54.488462 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.488466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.488531 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.488624 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.490164 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.490938 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.492394 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.492776 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.492894 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.496619 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.496623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.496625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.496654 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.498969 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.499832 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.499836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.499838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.499942 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.500463 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.500468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.500470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.500754 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.502496 (+ 1742us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.502602 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.502763 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":170,"threads_started":1}
I20260806 04:39:54.503276 7544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.487380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:44279 (local address 127.31.166.129:51637)
0806 04:39:54.487665 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.487683 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.488237 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.488826 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.488833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.489357 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.490002 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.490017 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.491136 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.491142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.492255 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.492265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.492967 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.492972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.493265 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.494083 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.494113 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.496373 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.499160 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.499167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.499186 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.499672 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.500070 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.500074 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.500076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.500314 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.500900 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.500907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.501351 (+ 444us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.501561 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.501963 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":153,"spinlock_wait_cycles":5632,"thread_start_us":110,"threads_started":1}
I20260806 04:39:54.503744 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:35665
--local_ip_for_outbound_sockets=127.31.166.130
--tserver_master_addrs=127.31.166.190:43117
--webserver_port=39977
--webserver_interface=127.31.166.130
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.130.keytab
--principal=oryx/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.166.190:43117,127.31.166.189:39575
--tserver_master_addrs=127.31.166.190:43117,127.31.166.189:39575,127.31.166.188:44279 with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:39:54.504149 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.486184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:39575 (local address 127.31.166.129:56449)
0806 04:39:54.486946 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.487026 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.488225 (+ 1199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.488924 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.488927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.489364 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.490015 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.490023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.491170 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.491173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.492461 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.492471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.494207 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.494211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.494340 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.494868 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.494890 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.497418 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.499855 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.499861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.499866 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.500128 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.500472 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.500478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.500483 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.500579 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.501112 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.501117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.501338 (+ 221us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.502201 (+ 863us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.502395 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":288,"spinlock_wait_cycles":24832,"thread_start_us":172,"threads_started":1}
I20260806 04:39:54.504218 7542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.486184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.129:45683)
0806 04:39:54.487159 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.487178 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.488225 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.488924 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.488927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.489360 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.490014 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.490023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.491170 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.491173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.492464 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.492471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.493217 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.493221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.493332 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.494149 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.494170 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.497307 (+ 3137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.500520 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.500525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.500527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.500763 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.501140 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.501143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.501145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.501244 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.501758 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.501761 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.501858 (+ 97us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.503713 (+ 1855us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.503908 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":299,"spinlock_wait_cycles":64768,"thread_start_us":160,"threads_started":1}
I20260806 04:39:54.505424 7538 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43117
I20260806 04:39:54.505672 7538 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:54.505856 7540 heartbeater.cc:347] Connected to a master server at 127.31.166.189:39575
I20260806 04:39:54.505926 7540 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:54.506171 7539 heartbeater.cc:347] Connected to a master server at 127.31.166.188:44279
I20260806 04:39:54.506235 7539 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:54.506330 7540 heartbeater.cc:511] Master 127.31.166.189:39575 requested a full tablet report, sending...
I20260806 04:39:54.506402 7539 heartbeater.cc:511] Master 127.31.166.188:44279 requested a full tablet report, sending...
I20260806 04:39:54.506965 7538 heartbeater.cc:511] Master 127.31.166.190:43117 requested a full tablet report, sending...
I20260806 04:39:54.507943 6488 ts_manager.cc:194] Re-registered known tserver with Master: 0c2609e68bce4b6c8edc78b221bd9f38 (127.31.166.129:39873)
I20260806 04:39:54.508299 7346 ts_manager.cc:194] Registered new tserver with Master: 0c2609e68bce4b6c8edc78b221bd9f38 (127.31.166.129:39873)
I20260806 04:39:54.508289 7149 ts_manager.cc:194] Registered new tserver with Master: 0c2609e68bce4b6c8edc78b221bd9f38 (127.31.166.129:39873)
I20260806 04:39:54.508869 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:45683
I20260806 04:39:54.562146 6991 heartbeater.cc:503] Master 127.31.166.190:43117 was elected leader, sending a full tablet report...
W20260806 04:39:54.626216 7546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:54.626487 7546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:54.626530 7546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:54.630578 7546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:54.630645 7546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:54.630704 7546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:39:54.636310 7546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.130.keytab
--principal=oryx/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:35665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=39977
--tserver_master_addrs=127.31.166.190:43117,127.31.166.189:39575,127.31.166.188:44279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:54.637318 7546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:54.638411 7546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:54.648978 7553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:54.648983 7552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:54.649035 7546 server_base.cc:1034] running on GCE node
W20260806 04:39:54.648978 7555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:54.649724 7546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:54.650391 7546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:54.651571 7546 hybrid_clock.cc:648] HybridClock initialized: now 1785991194651529 us; error 46 us; skew 500 ppm
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:54.654377 7546 init.cc:377] Logged in from keytab as oryx/127.31.166.130@KRBTEST.COM (short username oryx)
I20260806 04:39:54.655449 7546 webserver.cc:460] Webserver started at http://127.31.166.130:39977/ using document root <none> and password file <none>
I20260806 04:39:54.655992 7546 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:54.656068 7546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:54.659976 7546 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:39:54.662521 7563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:54.663873 7546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:39:54.663969 7546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "ee19bfd1525e40929731e010edb80f54"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "696f65b06e559dd5a58c141c77d2490d"
server_key_iv: "67f7ab38fb9af80ac0c09e2298e5c821"
server_key_version: "encryptionkey@0"
I20260806 04:39:54.664498 7546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:54.688738 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.685009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.31.166.188:44279)
0806 04:39:54.685147 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.685151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.685167 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:39:54.685296 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.685299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.685387 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.685510 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:54.685756 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.686294 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.687744 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.688215 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.688317 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.688435 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.688563 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.688633 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:39:54.688741 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.684904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:44279 (local address 127.0.0.1:49302)
0806 04:39:54.685056 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.685072 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.685179 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.685527 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.685529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.685545 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:54.685642 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.685648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.686402 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.686405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.687611 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.687622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.688332 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.688335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.688434 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.688460 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.688526 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.688600 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 04:39:54.689514 7356 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 04:39:54.689939 7386 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 824064c26018499f879fdf4970b0b8e4. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } } }
I20260806 04:39:54.690124 7386 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [sys.catalog]: This master's current role is: LEARNER
I20260806 04:39:54.693645 7546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:54.696578 7546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:54.696790 7546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:54.697417 7546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:54.698283 7546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:54.698329 7546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:54.698402 7546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:54.698433 7546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:54.707738 7546 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:35665
I20260806 04:39:54.707778 7685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:35665 every 8 connection(s)
I20260806 04:39:54.708827 7546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:39:54.712819 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 7546
I20260806 04:39:54.713027 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 6847
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.189@KRBTEST.COM
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.188@KRBTEST.COM
I20260806 04:39:54.724475 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:37535
--local_ip_for_outbound_sockets=127.31.166.131
--tserver_master_addrs=127.31.166.190:43117
--webserver_port=39259
--webserver_interface=127.31.166.131
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.131.keytab
--principal=oryx/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.166.190:43117,127.31.166.189:39575
--tserver_master_addrs=127.31.166.190:43117,127.31.166.189:39575,127.31.166.188:44279 with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:39:54.728646 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.710818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:35333 (local address 127.31.166.190:43117)
0806 04:39:54.710941 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.710946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.712147 (+ 1201us) server_negotiation.cc:407] Connection header received
0806 04:39:54.713722 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.713729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.713906 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.714033 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.715799 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.716622 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.718630 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.718979 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.719069 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.723499 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.723503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.723506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.723535 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.725907 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.726622 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.726626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.726628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.726735 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.727225 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.727228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.727230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.727462 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.727555 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.728329 (+ 774us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.728469 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:39:54.728771 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.712447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:59501 (local address 127.31.166.189:39575)
0806 04:39:54.712610 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.712615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.712749 (+ 134us) server_negotiation.cc:407] Connection header received
0806 04:39:54.714327 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.714333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.714433 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.714523 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.715799 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.716621 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.718590 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.718973 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.719058 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.723265 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.723272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.723279 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.723318 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.725703 (+ 2385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.726684 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.726690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.726692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.726805 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.727252 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.727257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.727260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.727554 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.727640 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.728348 (+ 708us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.728504 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 04:39:54.729741 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.711173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:44279 (local address 127.31.166.130:58803)
0806 04:39:54.711992 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.712047 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.714308 (+ 2261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.714910 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.714917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.715226 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.715622 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.715645 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.716930 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.716937 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.719524 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.719540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.720174 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.720179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.720274 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.721715 (+ 1441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.721738 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.723392 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.725965 (+ 2573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.725972 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.725976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.726498 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.726845 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.726847 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.726849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.727107 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.727614 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.727621 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.728030 (+ 409us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.728250 (+ 220us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.728595 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":1255,"spinlock_wait_cycles":31872,"thread_start_us":108,"threads_started":1}
I20260806 04:39:54.729741 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.711167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.130:35333)
0806 04:39:54.711991 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.712046 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.713332 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.714190 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.714197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.714733 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.715627 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.715646 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.716800 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.716804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.718445 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.718456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.719271 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.719275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.720253 (+ 978us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.720823 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.720845 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.723312 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.726050 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.726055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.726059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.726498 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.726845 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.726858 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.726860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.727107 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.727614 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.727621 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.728030 (+ 409us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.728864 (+ 834us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.729009 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":990,"spinlock_wait_cycles":61184,"thread_start_us":240,"threads_started":1}
I20260806 04:39:54.730485 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.710673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:58803 (local address 127.31.166.188:44279)
0806 04:39:54.710862 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.710879 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.712184 (+ 1305us) server_negotiation.cc:407] Connection header received
0806 04:39:54.714519 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.714526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.714610 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.714688 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.715808 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.716763 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.719660 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.720039 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.720149 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.723536 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.723540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.723543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.723573 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.725842 (+ 2269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.726622 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.726626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.726628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.726736 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.727225 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.727228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.727230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.727462 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.727555 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.730190 (+ 2635us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.730337 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 04:39:54.731249 7686 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43117
I20260806 04:39:54.731484 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.712200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:39575 (local address 127.31.166.130:59501)
0806 04:39:54.712657 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.712669 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.714186 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.714650 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.714653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.714828 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.715627 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.715646 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.716800 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.716804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.718445 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.718456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.719271 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.719275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.719619 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.720401 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.720431 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.723002 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.725884 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.725894 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.725917 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.726545 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.726954 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.726959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.726962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.727123 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.727731 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.727734 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.728046 (+ 312us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.729481 (+ 1435us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.729677 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":382,"spinlock_wait_cycles":72320,"thread_start_us":84,"threads_started":1}
I20260806 04:39:54.731525 7686 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:54.731573 7687 heartbeater.cc:347] Connected to a master server at 127.31.166.188:44279
I20260806 04:39:54.731987 7687 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:54.732556 7686 heartbeater.cc:511] Master 127.31.166.190:43117 requested a full tablet report, sending...
I20260806 04:39:54.732607 7688 heartbeater.cc:347] Connected to a master server at 127.31.166.189:39575
I20260806 04:39:54.732702 7688 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:54.732961 7688 heartbeater.cc:511] Master 127.31.166.189:39575 requested a full tablet report, sending...
I20260806 04:39:54.732556 7687 heartbeater.cc:511] Master 127.31.166.188:44279 requested a full tablet report, sending...
I20260806 04:39:54.733739 6488 ts_manager.cc:194] Re-registered known tserver with Master: ee19bfd1525e40929731e010edb80f54 (127.31.166.130:35665)
I20260806 04:39:54.734155 7346 ts_manager.cc:194] Registered new tserver with Master: ee19bfd1525e40929731e010edb80f54 (127.31.166.130:35665)
I20260806 04:39:54.734297 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:35333
I20260806 04:39:54.734750 7149 ts_manager.cc:194] Registered new tserver with Master: ee19bfd1525e40929731e010edb80f54 (127.31.166.130:35665)
W20260806 04:39:54.850212 7693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:39:54.850514 7693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:39:54.850610 7693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:39:54.854619 7693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:39:54.854701 7693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:39:54.854785 7693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:39:54.860153 7693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.131.keytab
--principal=oryx/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:37535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=39259
--tserver_master_addrs=127.31.166.190:43117,127.31.166.189:39575,127.31.166.188:44279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:39:54.861295 7693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:39:54.862375 7693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:39:54.872232 7698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:39:54.872217 7701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:54.872282 7693 server_base.cc:1034] running on GCE node
W20260806 04:39:54.872217 7699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:39:54.872946 7693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:39:54.873564 7693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:39:54.874744 7693 hybrid_clock.cc:648] HybridClock initialized: now 1785991194874710 us; error 47 us; skew 500 ppm
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:39:54.877965 7693 init.cc:377] Logged in from keytab as oryx/127.31.166.131@KRBTEST.COM (short username oryx)
I20260806 04:39:54.878831 7693 webserver.cc:460] Webserver started at http://127.31.166.131:39259/ using document root <none> and password file <none>
I20260806 04:39:54.879423 7693 fs_manager.cc:359] Metadata directory not provided
I20260806 04:39:54.879503 7693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:39:54.883455 7693 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260806 04:39:54.885643 7709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:54.886942 7693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:39:54.887137 7693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "595843906abc4b7386232904fb4280de"
format_stamp: "Formatted at 2026-08-06 04:39:53 on dist-test-slave-vkjp"
server_key: "4d31df220cbac47ca3f58fc65b04f8d9"
server_key_iv: "159fccb22af25a486811b56d8949a3ed"
server_key_version: "encryptionkey@0"
I20260806 04:39:54.887696 7693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:39:54.902714 7693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:39:54.905664 7693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:39:54.905880 7693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:39:54.906456 7693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:39:54.907470 7693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:39:54.907541 7693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:54.907613 7693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:39:54.907665 7693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:39:54.917178 7693 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:37535
I20260806 04:39:54.917201 7833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:37535 every 8 connection(s)
I20260806 04:39:54.918165 7693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:39:54.922921 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 7693
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.189@KRBTEST.COM
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
Aug 06 04:39:54 dist-test-slave-vkjp krb5kdc[6441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991194, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.188@KRBTEST.COM
I20260806 04:39:54.936332 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.925567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49310 (local address 127.31.166.188:44279)
0806 04:39:54.925832 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.925837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.925855 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:39:54.925888 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.925890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.925925 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.926020 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.926900 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.927692 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.928860 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.929197 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.929255 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.931449 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.931453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.931456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.931485 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.933641 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.934273 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.934277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.934280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.934396 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.934735 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.934741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.934746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.935015 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.935230 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.935969 (+ 739us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.936119 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":97,"threads_started":1}
I20260806 04:39:54.937135 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.920053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:50007 (local address 127.31.166.190:43117)
0806 04:39:54.920213 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.920217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.921211 (+ 994us) server_negotiation.cc:407] Connection header received
0806 04:39:54.922454 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.922457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.922511 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.922566 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.925502 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.926262 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.928205 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.928543 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.928615 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.932493 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.932500 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.932504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.932541 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.934320 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.935032 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.935037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.935041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.935165 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.935632 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.935635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.935637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.935891 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.935988 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.936849 (+ 861us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.936971 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 04:39:54.937860 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.920065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:44613 (local address 127.31.166.188:44279)
0806 04:39:54.920204 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.920208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.921334 (+ 1126us) server_negotiation.cc:407] Connection header received
0806 04:39:54.922432 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.922436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.922495 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.922600 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.926437 (+ 3837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.927230 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.928298 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.928600 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.928675 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.933607 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.933611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.933615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.933647 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.935474 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.935967 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.935973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.935976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.936055 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.936649 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.936652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.936653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.936869 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.936940 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.937597 (+ 657us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.937719 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:39:54.937901 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.921717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:51729 (local address 127.31.166.189:39575)
0806 04:39:54.921831 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.921842 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.921856 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:39:54.922451 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.922455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.922511 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.922566 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.924529 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.925342 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.927938 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.928290 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.928380 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.932877 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.932882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.932886 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.932918 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.935110 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.935619 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.935622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.935624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.935705 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.936011 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.936018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.936021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.936245 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.936352 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.937648 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.937760 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260806 04:39:54.938400 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.920441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:44279 (local address 127.31.166.131:44613)
0806 04:39:54.921215 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.921229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.922216 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.924620 (+ 2404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.924623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.925828 (+ 1205us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.925950 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.925957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.927381 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.927384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.928162 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.928170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.928843 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.928847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.928958 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.930179 (+ 1221us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.930198 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.933442 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.935593 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.935599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.935603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.935856 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.936171 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.936173 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.936175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.936307 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.937021 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.937025 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.937124 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.937499 (+ 375us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.937642 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":3,"spinlock_wait_cycles":9728,"thread_start_us":215,"threads_started":1}
I20260806 04:39:54.938419 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.921505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:39575 (local address 127.31.166.131:51729)
0806 04:39:54.921753 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.921767 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.922215 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.922845 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.922853 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.923447 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.924355 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.924377 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.926434 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.926437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.927806 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.927815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.928533 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.928536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.928715 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.929396 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.929422 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.932713 (+ 3291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.935249 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.935255 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.935268 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.935509 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.935815 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.935817 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.935820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.935908 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.936397 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.936403 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.937382 (+ 979us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.937477 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.938239 (+ 762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":740,"spinlock_wait_cycles":6656,"thread_start_us":89,"threads_started":1}
I20260806 04:39:54.938771 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.920452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.131:50007)
0806 04:39:54.921058 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.921110 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.922215 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.922844 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.922853 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.923447 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:39:54.924355 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.924377 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.927177 (+ 2800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.927181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.928066 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.928076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.928770 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.928776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.928901 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.929636 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:39:54.929655 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:39:54.932236 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:39:54.934463 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.934470 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.934486 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.934899 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.935291 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:39:54.935295 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:39:54.935297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:39:54.935506 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:39:54.936205 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:39:54.936212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:39:54.936543 (+ 331us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.936776 (+ 233us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.937133 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":125,"spinlock_wait_cycles":30848,"thread_start_us":157,"threads_started":1}
I20260806 04:39:54.939782 7834 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43117
I20260806 04:39:54.939837 7836 heartbeater.cc:347] Connected to a master server at 127.31.166.189:39575
I20260806 04:39:54.940034 7836 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:54.940057 7835 heartbeater.cc:347] Connected to a master server at 127.31.166.188:44279
I20260806 04:39:54.940169 7835 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:54.940032 7834 heartbeater.cc:464] Registering TS with master...
I20260806 04:39:54.940815 7834 heartbeater.cc:511] Master 127.31.166.190:43117 requested a full tablet report, sending...
I20260806 04:39:54.940613 7835 heartbeater.cc:511] Master 127.31.166.188:44279 requested a full tablet report, sending...
I20260806 04:39:54.940613 7836 heartbeater.cc:511] Master 127.31.166.189:39575 requested a full tablet report, sending...
I20260806 04:39:54.942082 6488 ts_manager.cc:194] Re-registered known tserver with Master: 595843906abc4b7386232904fb4280de (127.31.166.131:37535)
I20260806 04:39:54.942739 7149 ts_manager.cc:194] Registered new tserver with Master: 595843906abc4b7386232904fb4280de (127.31.166.131:37535)
I20260806 04:39:54.942962 6488 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:50007
I20260806 04:39:54.943480 7346 ts_manager.cc:194] Registered new tserver with Master: 595843906abc4b7386232904fb4280de (127.31.166.131:37535)
I20260806 04:39:54.948122 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.925342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.31.166.190:43117)
0806 04:39:54.930853 (+ 5511us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.930858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.930872 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:39:54.930908 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.930910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.930959 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.931052 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.933003 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.939836 (+ 6833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.940961 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.941229 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.941360 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.943892 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.943895 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.943897 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.943911 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.945901 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.946732 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.946735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.946737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.946841 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.947215 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.947218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.947219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.947393 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.947487 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.947780 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.947922 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5433,"thread_start_us":80,"threads_started":1}
I20260806 04:39:54.949678 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.924832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.31.166.189:39575)
0806 04:39:54.932346 (+ 7514us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.932351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.932364 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:39:54.932400 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.932405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.932455 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.941939 (+ 9484us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:54.942060 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.942821 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.943943 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.944250 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.944403 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.946043 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:54.946046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:54.946047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:54.946068 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:54.947797 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.948270 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.948274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.948275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.948359 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:54.948648 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:54.948650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:54.948651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:54.948807 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:54.948915 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.949220 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.949400 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7417,"thread_start_us":92,"threads_started":1}
I20260806 04:39:54.951537 6488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:39:54.954905 6488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:39:54.957950 7854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: NON_VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: true } } }
I20260806 04:39:54.958084 7854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:54.961378 7855 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4: attempting to promote NON_VOTER ee8e60bd63314a43b4fea6f479518272 to VOTER
I20260806 04:39:54.961678 7855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } }
I20260806 04:39:54.963332 7159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 FOLLOWER]: Refusing update from remote peer 824064c26018499f879fdf4970b0b8e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 04:39:54.963492 7356 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 1 LEARNER]: Refusing update from remote peer 824064c26018499f879fdf4970b0b8e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260806 04:39:54.963903 7308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 04:39:54.964162 7308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 04:39:54.965809 7854 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ee8e60bd63314a43b4fea6f479518272 (127.31.166.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } } }
I20260806 04:39:54.966481 7159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ee8e60bd63314a43b4fea6f479518272 (127.31.166.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } } }
I20260806 04:39:54.966867 7356 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ee8e60bd63314a43b4fea6f479518272 (127.31.166.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } } }
I20260806 04:39:54.968299 7308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } } }
I20260806 04:39:54.968410 7308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:54.968786 7855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } } }
I20260806 04:39:54.968904 7855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:54.969007 7854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } } }
I20260806 04:39:54.969092 7854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:54.970003 7386 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } } }
I20260806 04:39:54.970145 7386 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:39:54.973423 7866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "824064c26018499f879fdf4970b0b8e4" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } } }
I20260806 04:39:54.973565 7866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:39:54.976955 7101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.971054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:35665 (local address 127.0.0.1:50392)
0806 04:39:54.971216 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.971236 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.971334 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.972515 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.972519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.972542 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:54.972682 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.972695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.974049 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.974053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.975236 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.975246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.976399 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.976404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.976564 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.976596 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.976713 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.976793 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":26}
I20260806 04:39:54.976989 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.971533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50392 (local address 127.31.166.130:35665)
0806 04:39:54.971956 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.971971 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.971998 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:39:54.972084 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.972090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.972321 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.972541 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:54.972805 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.973918 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.975389 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.976267 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.976388 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.976592 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.976741 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.976843 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":191,"threads_started":1}
I20260806 04:39:54.979328 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.971464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:39873 (local address 127.0.0.1:52728)
0806 04:39:54.971764 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.971782 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.971907 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.972972 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.972978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.973003 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:54.974784 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.974796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.976127 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.976131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.977543 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.977554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.978871 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.978874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.978963 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.978986 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.979056 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.979155 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":64,"thread_start_us":109,"threads_started":1}
I20260806 04:39:54.979331 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.972503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.31.166.131:37535)
0806 04:39:54.973804 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.973819 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.973840 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:39:54.973908 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.973912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.974088 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.974251 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:54.974765 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.975869 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.977536 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.978632 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.978812 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.978952 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.979030 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.979155 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1206,"thread_start_us":117,"threads_started":1}
I20260806 04:39:54.979838 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.971893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52728 (local address 127.31.166.129:39873)
0806 04:39:54.972258 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:54.972281 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:54.972310 (+ 29us) server_negotiation.cc:407] Connection header received
0806 04:39:54.972403 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:54.972408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:54.972597 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:54.972788 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:54.974934 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.975990 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.977734 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.978570 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.978688 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.978803 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:54.979602 (+ 799us) server_negotiation.cc:299] Negotiation successful
0806 04:39:54.979699 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":101,"threads_started":1}
I20260806 04:39:54.979336 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:54.972308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:37535 (local address 127.0.0.1:56274)
0806 04:39:54.973406 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:54.973422 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:54.973530 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:54.974394 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:54.974397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:54.974415 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:54.974502 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.974508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.976005 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.976009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.977362 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:54.977377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:54.978793 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:54.978798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:54.978896 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:54.978925 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 04:39:54.979000 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 04:39:54.979127 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"mutex_wait_us":4,"thread_start_us":612,"threads_started":1}
I20260806 04:39:54.980189 7615 tablet_service.cc:1467] Processing CreateTablet for tablet d8f429bd8b9142a79d3bb367ce5790f2 (DEFAULT_TABLE table=test-table [id=6562e467a03e45aab87fc99ffa6ffa70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:54.981385 7615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f429bd8b9142a79d3bb367ce5790f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:54.982620 7469 tablet_service.cc:1467] Processing CreateTablet for tablet d8f429bd8b9142a79d3bb367ce5790f2 (DEFAULT_TABLE table=test-table [id=6562e467a03e45aab87fc99ffa6ffa70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:54.982632 7760 tablet_service.cc:1467] Processing CreateTablet for tablet d8f429bd8b9142a79d3bb367ce5790f2 (DEFAULT_TABLE table=test-table [id=6562e467a03e45aab87fc99ffa6ffa70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:54.983559 7469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f429bd8b9142a79d3bb367ce5790f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:54.983631 7760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f429bd8b9142a79d3bb367ce5790f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:54.989885 7873 tablet_bootstrap.cc:492] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54: Bootstrap starting.
I20260806 04:39:54.991029 7874 tablet_bootstrap.cc:492] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38: Bootstrap starting.
I20260806 04:39:54.992882 7873 tablet_bootstrap.cc:654] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:54.993104 7874 tablet_bootstrap.cc:654] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:54.993858 7873 log.cc:824] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:54.994024 7874 log.cc:824] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:54.995865 7874 tablet_bootstrap.cc:492] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38: No bootstrap required, opened a new log
I20260806 04:39:54.995873 7873 tablet_bootstrap.cc:492] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54: No bootstrap required, opened a new log
I20260806 04:39:54.996068 7874 ts_tablet_manager.cc:1397] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:39:54.996078 7873 ts_tablet_manager.cc:1397] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:39:54.994649 7875 tablet_bootstrap.cc:492] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de: Bootstrap starting.
I20260806 04:39:54.999637 7874 raft_consensus.cc:348] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } }
I20260806 04:39:54.999929 7874 raft_consensus.cc:374] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:54.999989 7874 raft_consensus.cc:729] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2609e68bce4b6c8edc78b221bd9f38, State: Initialized, Role: FOLLOWER
I20260806 04:39:55.000105 7873 raft_consensus.cc:348] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } }
I20260806 04:39:55.000361 7873 raft_consensus.cc:374] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:55.000419 7873 raft_consensus.cc:729] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee19bfd1525e40929731e010edb80f54, State: Initialized, Role: FOLLOWER
I20260806 04:39:55.000475 7874 consensus_queue.cc:260] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } }
I20260806 04:39:55.000594 7875 tablet_bootstrap.cc:654] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:55.000900 7873 consensus_queue.cc:260] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } }
I20260806 04:39:55.001379 7538 heartbeater.cc:503] Master 127.31.166.190:43117 was elected leader, sending a full tablet report...
I20260806 04:39:55.001542 7875 log.cc:824] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de: Log is configured to *not* fsync() on all Append() calls
I20260806 04:39:55.001958 7873 ts_tablet_manager.cc:1428] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:39:55.004968 7686 heartbeater.cc:503] Master 127.31.166.190:43117 was elected leader, sending a full tablet report...
I20260806 04:39:55.005013 7874 ts_tablet_manager.cc:1428] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 04:39:55.007458 7875 tablet_bootstrap.cc:492] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de: No bootstrap required, opened a new log
I20260806 04:39:55.007684 7875 ts_tablet_manager.cc:1397] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.001s
I20260806 04:39:55.011677 7875 raft_consensus.cc:348] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } }
I20260806 04:39:55.011896 7875 raft_consensus.cc:374] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:55.011965 7875 raft_consensus.cc:729] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595843906abc4b7386232904fb4280de, State: Initialized, Role: FOLLOWER
I20260806 04:39:55.012414 7875 consensus_queue.cc:260] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } }
I20260806 04:39:55.013100 7834 heartbeater.cc:503] Master 127.31.166.190:43117 was elected leader, sending a full tablet report...
I20260806 04:39:55.013419 7875 ts_tablet_manager.cc:1428] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260806 04:39:55.034895 7879 raft_consensus.cc:482] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:39:55.035141 7879 raft_consensus.cc:504] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } }
I20260806 04:39:55.036294 7879 leader_election.cc:288] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c2609e68bce4b6c8edc78b221bd9f38 (127.31.166.129:39873), 595843906abc4b7386232904fb4280de (127.31.166.131:37535)
I20260806 04:39:55.040347 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:55.036555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:48831 (local address 127.31.166.129:39873)
0806 04:39:55.036680 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:55.036686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:55.036702 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:39:55.036856 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:55.036861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:55.036920 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:55.036986 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:55.037330 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.037996 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.039319 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.039763 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.039911 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:55.039963 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:55.040159 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 04:39:55.040231 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:39:55.040351 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:55.036474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:39873 (local address 127.31.166.130:48831)
0806 04:39:55.036615 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:55.036631 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:55.036726 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:55.037022 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:55.037026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:55.037045 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:55.037197 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:55.037205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.038133 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.038136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:55.039172 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:55.039182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.039888 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.039893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:55.039998 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:55.040031 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 04:39:55.040134 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 04:39:55.040211 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":58,"spinlock_wait_cycles":2048}
I20260806 04:39:55.040427 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:55.036831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:45089 (local address 127.31.166.131:37535)
0806 04:39:55.036978 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:55.036985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:55.037000 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:39:55.037046 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:55.037049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:55.037090 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:55.037172 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:55.037483 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.037989 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.039280 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.039834 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.039949 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:55.040000 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:55.040229 (+ 229us) server_negotiation.cc:299] Negotiation successful
0806 04:39:55.040299 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:39:55.040462 7691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:55.036551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:37535 (local address 127.31.166.130:45089)
0806 04:39:55.036629 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:55.036641 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:55.036726 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:55.037194 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:55.037196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:55.037213 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:55.037336 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:55.037343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.038098 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.038102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:55.039160 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:55.039169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.040029 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.040032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:55.040104 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:55.040120 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 04:39:55.040209 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 04:39:55.040289 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":52,"spinlock_wait_cycles":9856}
I20260806 04:39:55.041065 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f429bd8b9142a79d3bb367ce5790f2" candidate_uuid: "ee19bfd1525e40929731e010edb80f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" is_pre_election: true
I20260806 04:39:55.041384 7491 raft_consensus.cc:2454] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee19bfd1525e40929731e010edb80f54 in term 0.
I20260806 04:39:55.041644 7784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f429bd8b9142a79d3bb367ce5790f2" candidate_uuid: "ee19bfd1525e40929731e010edb80f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595843906abc4b7386232904fb4280de" is_pre_election: true
I20260806 04:39:55.041883 7567 leader_election.cc:302] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c2609e68bce4b6c8edc78b221bd9f38, ee19bfd1525e40929731e010edb80f54; no voters:
I20260806 04:39:55.041942 7784 raft_consensus.cc:2454] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee19bfd1525e40929731e010edb80f54 in term 0.
I20260806 04:39:55.042199 7879 raft_consensus.cc:2781] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:39:55.042353 7879 raft_consensus.cc:482] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:39:55.042419 7879 raft_consensus.cc:3037] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:55.043519 7879 raft_consensus.cc:504] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } }
I20260806 04:39:55.043906 7879 leader_election.cc:288] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [CANDIDATE]: Term 1 election: Requested vote from peers 0c2609e68bce4b6c8edc78b221bd9f38 (127.31.166.129:39873), 595843906abc4b7386232904fb4280de (127.31.166.131:37535)
I20260806 04:39:55.044272 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f429bd8b9142a79d3bb367ce5790f2" candidate_uuid: "ee19bfd1525e40929731e010edb80f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2609e68bce4b6c8edc78b221bd9f38"
I20260806 04:39:55.044293 7784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f429bd8b9142a79d3bb367ce5790f2" candidate_uuid: "ee19bfd1525e40929731e010edb80f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "595843906abc4b7386232904fb4280de"
I20260806 04:39:55.044426 7491 raft_consensus.cc:3037] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:55.044435 7784 raft_consensus.cc:3037] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:55.045558 7491 raft_consensus.cc:2454] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee19bfd1525e40929731e010edb80f54 in term 1.
I20260806 04:39:55.045627 7784 raft_consensus.cc:2454] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee19bfd1525e40929731e010edb80f54 in term 1.
I20260806 04:39:55.045858 7567 leader_election.cc:302] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c2609e68bce4b6c8edc78b221bd9f38, ee19bfd1525e40929731e010edb80f54; no voters:
I20260806 04:39:55.046052 7879 raft_consensus.cc:2781] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:55.046288 7879 raft_consensus.cc:686] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [term 1 LEADER]: Becoming Leader. State: Replica: ee19bfd1525e40929731e010edb80f54, State: Running, Role: LEADER
I20260806 04:39:55.046643 7879 consensus_queue.cc:237] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } }
I20260806 04:39:55.050242 6487 catalog_manager.cc:5495] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee19bfd1525e40929731e010edb80f54 (127.31.166.130). New cstate: current_term: 1 leader_uuid: "ee19bfd1525e40929731e010edb80f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } health_report { overall_health: UNKNOWN } } }
I20260806 04:39:55.075496 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:55.072054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.31.166.130:35665)
0806 04:39:55.072189 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:55.072193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:55.072206 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:39:55.072326 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:55.072329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:55.072388 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:55.072475 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:55.072689 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.073185 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.074102 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.074305 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.074381 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:55.074682 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:55.075141 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:55.075232 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:55.075323 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 04:39:55.075373 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:39:55.081563 7784 raft_consensus.cc:1261] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de [term 1 FOLLOWER]: Refusing update from remote peer ee19bfd1525e40929731e010edb80f54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:55.081578 7491 raft_consensus.cc:1261] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 1 FOLLOWER]: Refusing update from remote peer ee19bfd1525e40929731e010edb80f54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:55.082355 7883 consensus_queue.cc:1035] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:55.082590 7879 consensus_queue.cc:1035] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:55.091455 7888 mvcc.cc:205] Tried to move back new op lower bound from 7315419935046590464 to 7315419934914232320. Current Snapshot: MvccSnapshot[applied={T|T < 7315419935046590464}]
I20260806 04:39:55.102451 6487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:55.102591 6487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:55.107033 6487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 824064c26018499f879fdf4970b0b8e4: Sending DeleteTablet for 3 replicas of tablet d8f429bd8b9142a79d3bb367ce5790f2
I20260806 04:39:55.108013 7760 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f429bd8b9142a79d3bb367ce5790f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:55 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:56274
I20260806 04:39:55.108011 7615 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f429bd8b9142a79d3bb367ce5790f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:55 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:50392
I20260806 04:39:55.108174 7469 tablet_service.cc:1514] Processing DeleteTablet for tablet d8f429bd8b9142a79d3bb367ce5790f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:55 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:52728
I20260806 04:39:55.108639 7899 tablet_replica.cc:330] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38: stopping tablet replica
I20260806 04:39:55.109328 7899 raft_consensus.cc:2229] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:55.109679 7899 raft_consensus.cc:2258] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:55.110095 7898 tablet_replica.cc:330] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de: stopping tablet replica
I20260806 04:39:55.110239 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 6455
I20260806 04:39:55.112216 7897 tablet_replica.cc:330] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54: stopping tablet replica
I20260806 04:39:55.112578 7897 raft_consensus.cc:2229] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:39:55.112955 7897 raft_consensus.cc:2258] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:55.114216 7899 ts_tablet_manager.cc:1905] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:55.115329 7898 raft_consensus.cc:2229] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:55.116648 7897 ts_tablet_manager.cc:1905] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:55.117054 7898 raft_consensus.cc:2258] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:55.118613 7898 ts_tablet_manager.cc:1905] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:55.119958 7897 ts_tablet_manager.cc:1918] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:39:55.120112 7897 log.cc:1196] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/d8f429bd8b9142a79d3bb367ce5790f2
I20260806 04:39:55.120548 7897 ts_tablet_manager.cc:1939] T d8f429bd8b9142a79d3bb367ce5790f2 P ee19bfd1525e40929731e010edb80f54: Deleting consensus metadata
I20260806 04:39:55.121615 7899 ts_tablet_manager.cc:1918] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:39:55.121721 7899 log.cc:1196] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-0/wal/wals/d8f429bd8b9142a79d3bb367ce5790f2
I20260806 04:39:55.122133 7899 ts_tablet_manager.cc:1939] T d8f429bd8b9142a79d3bb367ce5790f2 P 0c2609e68bce4b6c8edc78b221bd9f38: Deleting consensus metadata
W20260806 04:39:55.123260 7417 connection.cc:441] server connection from 127.0.0.1:52728 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52728 (request call id 1) could send its response
W20260806 04:39:55.123941 7566 connection.cc:441] server connection from 127.0.0.1:50392 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50392 (request call id 1) could send its response
I20260806 04:39:55.123963 7898 ts_tablet_manager.cc:1918] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:39:55.124053 7898 log.cc:1196] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/d8f429bd8b9142a79d3bb367ce5790f2
I20260806 04:39:55.124614 7898 ts_tablet_manager.cc:1939] T d8f429bd8b9142a79d3bb367ce5790f2 P 595843906abc4b7386232904fb4280de: Deleting consensus metadata
W20260806 04:39:55.125877 7711 connection.cc:441] server connection from 127.0.0.1:56274 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56274 (request call id 1) could send its response
I20260806 04:39:55.131237 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:55.109191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.31.166.189:39575)
0806 04:39:55.109294 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:55.109298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:55.119999 (+ 10701us) server_negotiation.cc:407] Connection header received
0806 04:39:55.120244 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:55.120250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:55.120333 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:55.120394 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:55.121636 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.122440 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.123639 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.124048 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.124147 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:55.126591 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:55.126595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:55.126599 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:55.126626 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:55.128598 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:55.129092 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:55.129097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:55.129099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:55.129186 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:55.129490 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:55.129493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:55.129495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:55.129691 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:55.129803 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:55.130604 (+ 801us) server_negotiation.cc:299] Negotiation successful
0806 04:39:55.131038 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 04:39:55.136543 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:55.124347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.31.166.189:39575)
0806 04:39:55.124466 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:55.124470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:55.124931 (+ 461us) server_negotiation.cc:407] Connection header received
0806 04:39:55.125096 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:55.125100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:55.125140 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:55.125187 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:55.126852 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.127646 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.129880 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.130216 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.130285 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:55.133023 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:55.133027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:55.133030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:55.133054 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:55.134692 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:55.135234 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:55.135237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:55.135239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:55.135322 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:55.135615 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:55.135618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:55.135619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:55.135830 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:55.135932 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:55.136283 (+ 351us) server_negotiation.cc:299] Negotiation successful
0806 04:39:55.136393 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 04:39:55.137965 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:55.125228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.31.166.188:44279)
0806 04:39:55.125344 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:55.125349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:55.126795 (+ 1446us) server_negotiation.cc:407] Connection header received
0806 04:39:55.127004 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:55.127008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:55.127068 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:55.127154 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:55.128094 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.128893 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.130050 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.130431 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.130513 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:55.133307 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:55.133311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:55.133314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:55.133340 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:55.135499 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:55.136024 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:55.136027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:55.136029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:55.136118 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:55.136453 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:55.136457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:55.136460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:55.136718 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:55.136823 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:55.137608 (+ 785us) server_negotiation.cc:299] Negotiation successful
0806 04:39:55.137763 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 04:39:55.140146 7545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:55.109542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.31.166.188:44279)
0806 04:39:55.109639 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:55.109644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:55.130740 (+ 21096us) server_negotiation.cc:407] Connection header received
0806 04:39:55.130780 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:55.130785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:55.130836 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:55.130914 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:39:55.132071 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.132839 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.134061 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:55.134429 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:55.134521 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:55.136199 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:39:55.136202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:39:55.136206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:39:55.136230 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:39:55.138313 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:55.138843 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:55.138847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:55.138849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:55.138942 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:39:55.139275 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:39:55.139281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:39:55.139286 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:39:55.139487 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:39:55.139575 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:55.139886 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 04:39:55.140000 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 04:39:56.094141 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.093620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.131:36851)
0806 04:39:56.093937 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:56.094035 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":81,"threads_started":1}
W20260806 04:39:56.094633 7834 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:43117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
I20260806 04:39:56.097322 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.096906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.129:51427)
0806 04:39:56.097129 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:56.097222 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":84,"threads_started":1}
W20260806 04:39:56.097661 7538 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:43117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
I20260806 04:39:56.098770 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.098260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.130:50099)
0806 04:39:56.098568 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:56.098656 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
W20260806 04:39:56.099164 7686 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:43117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
I20260806 04:39:56.607960 7914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 824064c26018499f879fdf4970b0b8e4)
I20260806 04:39:56.607960 7913 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 824064c26018499f879fdf4970b0b8e4)
I20260806 04:39:56.608266 7913 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } }
I20260806 04:39:56.608265 7914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } }
I20260806 04:39:56.609509 7913 leader_election.cc:288] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 824064c26018499f879fdf4970b0b8e4 (127.31.166.190:43117), 37c687289f6c45a785cb45ca13c8f276 (127.31.166.189:39575)
I20260806 04:39:56.609586 7914 leader_election.cc:288] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 824064c26018499f879fdf4970b0b8e4 (127.31.166.190:43117), ee8e60bd63314a43b4fea6f479518272 (127.31.166.188:44279)
I20260806 04:39:56.610311 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.609697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50734)
0806 04:39:56.610042 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:56.610142 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":91,"threads_started":1}
W20260806 04:39:56.610599 7332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
I20260806 04:39:56.610877 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.609697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50732)
0806 04:39:56.610686 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:56.610771 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":916,"thread_start_us":76,"threads_started":1}
W20260806 04:39:56.611124 7133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
I20260806 04:39:56.611696 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.611402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50758)
0806 04:39:56.611563 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:56.611622 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260806 04:39:56.612073 7332 leader_election.cc:334] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 824064c26018499f879fdf4970b0b8e4 (127.31.166.190:43117): Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
I20260806 04:39:56.612143 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.611914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50760)
0806 04:39:56.612031 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:56.612083 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 04:39:56.612291 7133 leader_election.cc:334] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 824064c26018499f879fdf4970b0b8e4 (127.31.166.190:43117): Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
I20260806 04:39:56.614410 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.610208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.31.166.188:44279)
0806 04:39:56.610447 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:56.610452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:56.610974 (+ 522us) server_negotiation.cc:407] Connection header received
0806 04:39:56.611028 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:56.611031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:56.611102 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:56.611184 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:56.611580 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:56.612340 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:56.613418 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:56.613836 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:56.613967 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:56.614020 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:56.614208 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 04:39:56.614282 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":55,"threads_started":1}
I20260806 04:39:56.614436 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.609856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:44279 (local address 127.0.0.1:49342)
0806 04:39:56.610728 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:56.610741 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:56.610875 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:56.611316 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:56.611319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:56.611341 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:56.611446 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:56.611453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:56.612495 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:56.612499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:56.613277 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:56.613286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:56.613963 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:56.613967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:56.614081 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:56.614100 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:39:56.614189 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 04:39:56.614267 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"thread_start_us":80,"threads_started":1}
I20260806 04:39:56.614681 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.610135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.31.166.189:39575)
0806 04:39:56.610457 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:56.610464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:56.610606 (+ 142us) server_negotiation.cc:407] Connection header received
0806 04:39:56.610875 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:56.610882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:56.611057 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:56.611178 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:56.611664 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:56.612590 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:56.613656 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:56.614082 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:56.614218 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:56.614272 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:56.614490 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 04:39:56.614563 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":71,"threads_started":1}
I20260806 04:39:56.614681 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.609856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:39575 (local address 127.0.0.1:60702)
0806 04:39:56.610364 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:56.610489 (+ 125us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:56.610637 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:56.611303 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:56.611306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:56.611330 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:56.611458 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:56.611467 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:56.612710 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:56.612717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:56.613511 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:56.613521 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:56.614213 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:56.614218 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:56.614331 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:56.614350 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:39:56.614470 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 04:39:56.614548 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":70,"threads_started":1}
I20260806 04:39:56.615166 7356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37c687289f6c45a785cb45ca13c8f276" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ee8e60bd63314a43b4fea6f479518272" is_pre_election: true
I20260806 04:39:56.615166 7159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee8e60bd63314a43b4fea6f479518272" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "37c687289f6c45a785cb45ca13c8f276" is_pre_election: true
I20260806 04:39:56.615409 7159 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee8e60bd63314a43b4fea6f479518272 in term 1.
I20260806 04:39:56.615409 7356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37c687289f6c45a785cb45ca13c8f276 in term 1.
I20260806 04:39:56.615801 7130 leader_election.cc:302] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37c687289f6c45a785cb45ca13c8f276, ee8e60bd63314a43b4fea6f479518272; no voters: 824064c26018499f879fdf4970b0b8e4
I20260806 04:39:56.615782 7329 leader_election.cc:302] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37c687289f6c45a785cb45ca13c8f276, ee8e60bd63314a43b4fea6f479518272; no voters: 824064c26018499f879fdf4970b0b8e4
I20260806 04:39:56.616022 7913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:39:56.616097 7913 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 824064c26018499f879fdf4970b0b8e4)
I20260806 04:39:56.616155 7913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:39:56.616176 7914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:39:56.616246 7914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 824064c26018499f879fdf4970b0b8e4)
I20260806 04:39:56.616303 7914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:39:56.617250 7913 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } }
I20260806 04:39:56.617359 7914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } }
I20260806 04:39:56.617681 7913 leader_election.cc:288] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [CANDIDATE]: Term 2 election: Requested vote from peers 824064c26018499f879fdf4970b0b8e4 (127.31.166.190:43117), 37c687289f6c45a785cb45ca13c8f276 (127.31.166.189:39575)
I20260806 04:39:56.617864 7914 leader_election.cc:288] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [CANDIDATE]: Term 2 election: Requested vote from peers 824064c26018499f879fdf4970b0b8e4 (127.31.166.190:43117), ee8e60bd63314a43b4fea6f479518272 (127.31.166.188:44279)
I20260806 04:39:56.618024 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.617789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50762)
0806 04:39:56.617901 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:56.617960 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 04:39:56.618049 7159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee8e60bd63314a43b4fea6f479518272" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "37c687289f6c45a785cb45ca13c8f276"
I20260806 04:39:56.618223 7159 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ee8e60bd63314a43b4fea6f479518272 in current term 2: Already voted for candidate 37c687289f6c45a785cb45ca13c8f276 in this term.
I20260806 04:39:56.618363 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.618098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50768)
0806 04:39:56.618237 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:56.618291 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 04:39:56.618649 7356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37c687289f6c45a785cb45ca13c8f276" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ee8e60bd63314a43b4fea6f479518272"
I20260806 04:39:56.618850 7356 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 37c687289f6c45a785cb45ca13c8f276 in current term 2: Already voted for candidate ee8e60bd63314a43b4fea6f479518272 in this term.
I20260806 04:39:56.619064 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.618799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50770)
0806 04:39:56.618941 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:56.618994 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260806 04:39:56.619175 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:56.618908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50782)
0806 04:39:56.619011 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:56.619073 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260806 04:39:56.619235 7332 leader_election.cc:334] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 824064c26018499f879fdf4970b0b8e4 (127.31.166.190:43117): Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
W20260806 04:39:56.619278 7133 leader_election.cc:334] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 824064c26018499f879fdf4970b0b8e4 (127.31.166.190:43117): Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
I20260806 04:39:56.619314 7332 leader_election.cc:302] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee8e60bd63314a43b4fea6f479518272; no voters: 37c687289f6c45a785cb45ca13c8f276, 824064c26018499f879fdf4970b0b8e4
I20260806 04:39:56.619378 7133 leader_election.cc:302] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37c687289f6c45a785cb45ca13c8f276; no voters: 824064c26018499f879fdf4970b0b8e4, ee8e60bd63314a43b4fea6f479518272
I20260806 04:39:56.619451 7913 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:39:56.619585 7914 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:39:57.095882 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:57.095361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.131:47929)
0806 04:39:57.095708 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:57.095776 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":96,"threads_started":1}
I20260806 04:39:57.098618 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:57.098191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.129:41647)
0806 04:39:57.098490 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:57.098544 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":89,"threads_started":1}
I20260806 04:39:57.100317 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:57.099750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.130:56405)
0806 04:39:57.100126 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:57.100197 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":55,"threads_started":1}
I20260806 04:39:58.097415 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.096909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.131:56629)
0806 04:39:58.097222 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.097295 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":82,"threads_started":1}
I20260806 04:39:58.099992 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.099510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.129:43065)
0806 04:39:58.099823 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.099890 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":92,"threads_started":1}
I20260806 04:39:58.101649 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.101079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.130:42765)
0806 04:39:58.101524 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.101575 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":81,"threads_started":1}
I20260806 04:39:58.344220 7931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:39:58.344381 7931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } }
I20260806 04:39:58.344990 7931 leader_election.cc:288] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 824064c26018499f879fdf4970b0b8e4 (127.31.166.190:43117), ee8e60bd63314a43b4fea6f479518272 (127.31.166.188:44279)
I20260806 04:39:58.345583 7356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37c687289f6c45a785cb45ca13c8f276" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ee8e60bd63314a43b4fea6f479518272" is_pre_election: true
I20260806 04:39:58.345749 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.345219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:50996)
0806 04:39:58.345524 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.345594 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":85,"threads_started":1}
I20260806 04:39:58.345738 7356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37c687289f6c45a785cb45ca13c8f276 in term 2.
I20260806 04:39:58.346231 7130 leader_election.cc:302] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37c687289f6c45a785cb45ca13c8f276, ee8e60bd63314a43b4fea6f479518272; no voters:
I20260806 04:39:58.346449 7931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:39:58.346522 7931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:39:58.346582 7931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:39:58.346858 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.346632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:51004)
0806 04:39:58.346740 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.346799 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260806 04:39:58.346997 7133 leader_election.cc:334] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 824064c26018499f879fdf4970b0b8e4 (127.31.166.190:43117): Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
I20260806 04:39:58.348119 7931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } }
I20260806 04:39:58.348505 7931 leader_election.cc:288] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [CANDIDATE]: Term 3 election: Requested vote from peers 824064c26018499f879fdf4970b0b8e4 (127.31.166.190:43117), ee8e60bd63314a43b4fea6f479518272 (127.31.166.188:44279)
I20260806 04:39:58.349018 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.348711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:51008)
0806 04:39:58.348838 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.348935 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 04:39:58.348976 7356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37c687289f6c45a785cb45ca13c8f276" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ee8e60bd63314a43b4fea6f479518272"
I20260806 04:39:58.349118 7356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:39:58.349931 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.349524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:51020)
0806 04:39:58.349817 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.349869 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234}
W20260806 04:39:58.350049 7133 leader_election.cc:334] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 824064c26018499f879fdf4970b0b8e4 (127.31.166.190:43117): Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
I20260806 04:39:58.350538 7356 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37c687289f6c45a785cb45ca13c8f276 in term 3.
I20260806 04:39:58.350872 7130 leader_election.cc:302] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37c687289f6c45a785cb45ca13c8f276, ee8e60bd63314a43b4fea6f479518272; no voters: 824064c26018499f879fdf4970b0b8e4
I20260806 04:39:58.351308 7931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:39:58.351531 7931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [term 3 LEADER]: Becoming Leader. State: Replica: 37c687289f6c45a785cb45ca13c8f276, State: Running, Role: LEADER
I20260806 04:39:58.351806 7931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } }
I20260806 04:39:58.352946 7934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37c687289f6c45a785cb45ca13c8f276. Latest consensus state: current_term: 3 leader_uuid: "37c687289f6c45a785cb45ca13c8f276" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } } }
I20260806 04:39:58.353096 7934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:58.353482 7937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:39:58.355976 7937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:39:58.356427 7937 catalog_manager.cc:1259] Loaded cluster ID: b20d78810fde47c7ae5a4e7969b72dc4
I20260806 04:39:58.356479 7937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:39:58.357087 7937 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:39:58.357524 7937 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: Loaded TSK: 0
I20260806 04:39:58.358242 7937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:39:58.434566 7149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:39:58.437834 7149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:39:58.441826 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.441478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:51036)
0806 04:39:58.441629 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.441707 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 04:39:58.441797 7356 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [term 3 FOLLOWER]: Refusing update from remote peer 37c687289f6c45a785cb45ca13c8f276: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260806 04:39:58.442317 7934 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 04:39:58.442595 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.442336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.0.0.1:51050)
0806 04:39:58.442464 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.442528 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 04:39:58.442714 7133 consensus_peers.cc:487] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 -> Peer 824064c26018499f879fdf4970b0b8e4 (127.31.166.190:43117): Couldn't send request to peer 824064c26018499f879fdf4970b0b8e4. Status: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 04:39:58.444123 7940 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37c687289f6c45a785cb45ca13c8f276. Latest consensus state: current_term: 3 leader_uuid: "37c687289f6c45a785cb45ca13c8f276" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } } }
I20260806 04:39:58.444242 7940 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:39:58.445824 7931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "37c687289f6c45a785cb45ca13c8f276" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } } }
I20260806 04:39:58.446007 7931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276 [sys.catalog]: This master's current role is: LEADER
I20260806 04:39:58.446108 7940 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "37c687289f6c45a785cb45ca13c8f276" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "824064c26018499f879fdf4970b0b8e4" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43117 } } peers { permanent_uuid: "37c687289f6c45a785cb45ca13c8f276" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 39575 } attrs { promote: false } } peers { permanent_uuid: "ee8e60bd63314a43b4fea6f479518272" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 44279 } attrs { promote: false } } }
I20260806 04:39:58.446276 7940 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee8e60bd63314a43b4fea6f479518272 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:39:58.446481 7941 mvcc.cc:205] Tried to move back new op lower bound from 7315419948811689984 to 7315419948451614720. Current Snapshot: MvccSnapshot[applied={T|T < 7315419948811689984}]
I20260806 04:39:58.457434 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.453459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.31.166.129:39873)
0806 04:39:58.453691 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:58.453702 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:58.453714 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:39:58.453758 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:58.453761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:58.453802 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:58.453905 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:58.454189 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.454765 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.456270 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.456865 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.456997 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:58.457054 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:58.457216 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 04:39:58.457295 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
I20260806 04:39:58.457434 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.452717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:39873 (local address 127.0.0.1:52740)
0806 04:39:58.453017 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.453032 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:58.453120 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:58.453924 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:58.453927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:58.453945 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:58.454054 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:58.454061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.454880 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.454884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:58.456123 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:58.456134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.456985 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.456989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:58.457078 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:58.457101 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:39:58.457200 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 04:39:58.457276 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":31,"thread_start_us":144,"threads_started":1}
I20260806 04:39:58.457943 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.452462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:37535 (local address 127.0.0.1:56280)
0806 04:39:58.452969 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.452985 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:58.453078 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:58.454049 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:58.454051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:58.454065 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:58.454174 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:58.454180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.455303 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.455306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:58.456570 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:58.456578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.457449 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.457453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:58.457586 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:58.457633 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 04:39:58.457696 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 04:39:58.457806 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":142,"spinlock_wait_cycles":3968}
I20260806 04:39:58.458096 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.453210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.31.166.130:35665)
0806 04:39:58.453498 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:58.453512 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:58.453722 (+ 210us) server_negotiation.cc:407] Connection header received
0806 04:39:58.453791 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:58.453794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:58.453851 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:58.453931 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:58.454360 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.455154 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.456565 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.457201 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.457331 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:58.457405 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:58.457868 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 04:39:58.457949 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":92,"threads_started":1}
I20260806 04:39:58.458477 7947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.452605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.31.166.131:37535)
0806 04:39:58.453828 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:58.453844 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:58.453857 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:39:58.453912 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:58.453917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:58.453964 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:58.454047 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:58.454303 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.455154 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.456686 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.457317 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.458209 (+ 892us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:58.458265 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:58.458304 (+ 39us) server_negotiation.cc:299] Negotiation successful
0806 04:39:58.458371 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1131,"thread_start_us":77,"threads_started":1}
I20260806 04:39:58.458429 7469 tablet_service.cc:1467] Processing CreateTablet for tablet 59054fa12c094bf7a0a3d2b33d2554d8 (DEFAULT_TABLE table=test-table [id=b92ed1b9d3f842e39d03eec45be3300f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:58.458745 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.453092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:35665 (local address 127.0.0.1:50420)
0806 04:39:58.453533 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.453546 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:58.453637 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:58.454060 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:58.454063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:58.454093 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:58.454202 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:58.454207 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.455264 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.455268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:58.456436 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:58.456445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.457452 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.457454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:58.457583 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:58.457631 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 04:39:58.457699 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 04:39:58.457804 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":120,"spinlock_wait_cycles":4608,"thread_start_us":91,"threads_started":1}
I20260806 04:39:58.458782 7469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59054fa12c094bf7a0a3d2b33d2554d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:58.458993 7760 tablet_service.cc:1467] Processing CreateTablet for tablet 59054fa12c094bf7a0a3d2b33d2554d8 (DEFAULT_TABLE table=test-table [id=b92ed1b9d3f842e39d03eec45be3300f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:58.459302 7760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59054fa12c094bf7a0a3d2b33d2554d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:58.459307 7615 tablet_service.cc:1467] Processing CreateTablet for tablet 59054fa12c094bf7a0a3d2b33d2554d8 (DEFAULT_TABLE table=test-table [id=b92ed1b9d3f842e39d03eec45be3300f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:39:58.459594 7615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59054fa12c094bf7a0a3d2b33d2554d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:39:58.461608 7953 tablet_bootstrap.cc:492] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38: Bootstrap starting.
I20260806 04:39:58.462128 7954 tablet_bootstrap.cc:492] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de: Bootstrap starting.
I20260806 04:39:58.462834 7953 tablet_bootstrap.cc:654] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:58.462917 7954 tablet_bootstrap.cc:654] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:58.463277 7955 tablet_bootstrap.cc:492] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54: Bootstrap starting.
I20260806 04:39:58.464176 7955 tablet_bootstrap.cc:654] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54: Neither blocks nor log segments found. Creating new log.
I20260806 04:39:58.464479 7954 tablet_bootstrap.cc:492] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de: No bootstrap required, opened a new log
I20260806 04:39:58.464582 7954 ts_tablet_manager.cc:1397] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 04:39:58.464850 7953 tablet_bootstrap.cc:492] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38: No bootstrap required, opened a new log
I20260806 04:39:58.464957 7953 ts_tablet_manager.cc:1397] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:39:58.464985 7954 raft_consensus.cc:348] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } }
I20260806 04:39:58.465090 7954 raft_consensus.cc:374] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:58.465117 7954 raft_consensus.cc:729] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 595843906abc4b7386232904fb4280de, State: Initialized, Role: FOLLOWER
I20260806 04:39:58.465255 7954 consensus_queue.cc:260] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } }
I20260806 04:39:58.465447 7953 raft_consensus.cc:348] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } }
I20260806 04:39:58.465574 7954 ts_tablet_manager.cc:1428] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:39:58.465575 7953 raft_consensus.cc:374] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:58.465629 7953 raft_consensus.cc:729] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c2609e68bce4b6c8edc78b221bd9f38, State: Initialized, Role: FOLLOWER
I20260806 04:39:58.465772 7953 consensus_queue.cc:260] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } }
I20260806 04:39:58.466140 7953 ts_tablet_manager.cc:1428] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260806 04:39:58.466446 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.466045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.131:44269)
0806 04:39:58.466304 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.466369 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260806 04:39:58.466831 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.466522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.129:37291)
0806 04:39:58.466714 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.466764 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260806 04:39:58.467413 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.467136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.131:48391)
0806 04:39:58.467289 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.467339 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 04:39:58.468101 7955 tablet_bootstrap.cc:492] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54: No bootstrap required, opened a new log
I20260806 04:39:58.468181 7955 ts_tablet_manager.cc:1397] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260806 04:39:58.468576 7955 raft_consensus.cc:348] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } }
I20260806 04:39:58.468704 7955 raft_consensus.cc:374] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:39:58.468740 7955 raft_consensus.cc:729] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee19bfd1525e40929731e010edb80f54, State: Initialized, Role: FOLLOWER
I20260806 04:39:58.468863 7955 consensus_queue.cc:260] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } }
I20260806 04:39:58.468914 7836 heartbeater.cc:503] Master 127.31.166.189:39575 was elected leader, sending a full tablet report...
I20260806 04:39:58.469311 7955 ts_tablet_manager.cc:1428] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:39:58.469964 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.469738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.130:57817)
0806 04:39:58.469842 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.469895 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260806 04:39:58.471355 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.470376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.129:60181)
0806 04:39:58.471228 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.471285 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":778}
I20260806 04:39:58.471663 7540 heartbeater.cc:503] Master 127.31.166.189:39575 was elected leader, sending a full tablet report...
I20260806 04:39:58.471911 7688 heartbeater.cc:503] Master 127.31.166.189:39575 was elected leader, sending a full tablet report...
I20260806 04:39:58.472620 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.472088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.130:32963)
0806 04:39:58.472526 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.472570 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385}
W20260806 04:39:58.488639 7541 tablet.cc:2369] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:39:58.507918 7960 raft_consensus.cc:482] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:39:58.508080 7960 raft_consensus.cc:504] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } }
I20260806 04:39:58.509250 7960 leader_election.cc:288] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c2609e68bce4b6c8edc78b221bd9f38 (127.31.166.129:39873), ee19bfd1525e40929731e010edb80f54 (127.31.166.130:35665)
I20260806 04:39:58.513154 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.509592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:39873 (local address 127.31.166.131:45771)
0806 04:39:58.509752 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.509771 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:58.509893 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:58.510213 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:58.510216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:58.510237 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:58.510387 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:58.510393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.511171 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.511178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:58.512031 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:58.512041 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.512713 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.512717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:58.512815 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:58.512838 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:39:58.512937 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 04:39:58.513012 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 04:39:58.513154 7952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.509703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:45771 (local address 127.31.166.129:39873)
0806 04:39:58.509834 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:58.509837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:58.509854 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:39:58.510031 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:58.510035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:58.510092 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:58.510196 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:58.510512 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.511030 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.512166 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.512568 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.512709 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:58.512770 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:58.512954 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 04:39:58.513030 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:39:58.513746 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59054fa12c094bf7a0a3d2b33d2554d8" candidate_uuid: "595843906abc4b7386232904fb4280de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" is_pre_election: true
I20260806 04:39:58.513916 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.509942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:35665 (local address 127.31.166.131:50519)
0806 04:39:58.510432 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.510444 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:39:58.510593 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:39:58.510875 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:39:58.510878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:39:58.510896 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:39:58.511023 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:58.511028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.511870 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.511875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:58.512675 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:39:58.512684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.513465 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.513469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:39:58.513580 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:58.513603 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:39:58.513701 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 04:39:58.513766 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":169,"threads_started":1}
I20260806 04:39:58.513928 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.510128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:50519 (local address 127.31.166.130:35665)
0806 04:39:58.510278 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:58.510284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:58.510540 (+ 256us) server_negotiation.cc:407] Connection header received
0806 04:39:58.510707 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:58.510712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:58.510763 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:58.510859 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:39:58.511184 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.511739 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.512811 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.513342 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.513478 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:58.513539 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:58.513721 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 04:39:58.513810 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 04:39:58.513955 7491 raft_consensus.cc:2454] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 595843906abc4b7386232904fb4280de in term 0.
I20260806 04:39:58.514451 7713 leader_election.cc:302] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c2609e68bce4b6c8edc78b221bd9f38, 595843906abc4b7386232904fb4280de; no voters:
I20260806 04:39:58.514462 7638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59054fa12c094bf7a0a3d2b33d2554d8" candidate_uuid: "595843906abc4b7386232904fb4280de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee19bfd1525e40929731e010edb80f54" is_pre_election: true
I20260806 04:39:58.514703 7638 raft_consensus.cc:2454] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 595843906abc4b7386232904fb4280de in term 0.
I20260806 04:39:58.514756 7960 raft_consensus.cc:2781] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:39:58.514827 7960 raft_consensus.cc:482] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:39:58.514883 7960 raft_consensus.cc:3037] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:58.516098 7960 raft_consensus.cc:504] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } }
I20260806 04:39:58.516453 7960 leader_election.cc:288] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [CANDIDATE]: Term 1 election: Requested vote from peers 0c2609e68bce4b6c8edc78b221bd9f38 (127.31.166.129:39873), ee19bfd1525e40929731e010edb80f54 (127.31.166.130:35665)
I20260806 04:39:58.516935 7491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59054fa12c094bf7a0a3d2b33d2554d8" candidate_uuid: "595843906abc4b7386232904fb4280de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c2609e68bce4b6c8edc78b221bd9f38"
I20260806 04:39:58.516978 7638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59054fa12c094bf7a0a3d2b33d2554d8" candidate_uuid: "595843906abc4b7386232904fb4280de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee19bfd1525e40929731e010edb80f54"
I20260806 04:39:58.517091 7491 raft_consensus.cc:3037] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:58.517198 7638 raft_consensus.cc:3037] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:39:58.518250 7638 raft_consensus.cc:2454] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 595843906abc4b7386232904fb4280de in term 1.
I20260806 04:39:58.518249 7491 raft_consensus.cc:2454] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 595843906abc4b7386232904fb4280de in term 1.
I20260806 04:39:58.518538 7713 leader_election.cc:302] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 595843906abc4b7386232904fb4280de, ee19bfd1525e40929731e010edb80f54; no voters:
I20260806 04:39:58.518720 7960 raft_consensus.cc:2781] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:39:58.519016 7960 raft_consensus.cc:686] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [term 1 LEADER]: Becoming Leader. State: Replica: 595843906abc4b7386232904fb4280de, State: Running, Role: LEADER
I20260806 04:39:58.519553 7960 consensus_queue.cc:237] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } } peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } }
I20260806 04:39:58.519784 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.519512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.131:50299)
0806 04:39:58.519625 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.519681 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 04:39:58.522595 7147 catalog_manager.cc:5495] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de reported cstate change: term changed from 0 to 1, leader changed from <none> to 595843906abc4b7386232904fb4280de (127.31.166.131). New cstate: current_term: 1 leader_uuid: "595843906abc4b7386232904fb4280de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "595843906abc4b7386232904fb4280de" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 37535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 } health_report { overall_health: UNKNOWN } } }
I20260806 04:39:58.539723 7947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.536150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56290 (local address 127.31.166.131:37535)
0806 04:39:58.536294 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:39:58.536298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:39:58.536311 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:39:58.536413 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:39:58.536416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:39:58.536473 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:39:58.536563 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:39:58.536776 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.537268 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.538140 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:39:58.538354 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:39:58.538465 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:39:58.538777 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:39:58.539331 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:39:58.539452 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 04:39:58.539519 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 04:39:58.539572 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:39:58.542773 7491 raft_consensus.cc:1261] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 1 FOLLOWER]: Refusing update from remote peer 595843906abc4b7386232904fb4280de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:58.542968 7638 raft_consensus.cc:1261] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54 [term 1 FOLLOWER]: Refusing update from remote peer 595843906abc4b7386232904fb4280de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:39:58.543395 7960 consensus_queue.cc:1035] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c2609e68bce4b6c8edc78b221bd9f38" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:58.543609 7965 consensus_queue.cc:1035] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee19bfd1525e40929731e010edb80f54" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:39:58.545761 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.544840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.129:52967)
0806 04:39:58.545613 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.545681 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":691}
I20260806 04:39:58.546375 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.546060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.130:35831)
0806 04:39:58.546244 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.546300 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260806 04:39:58.549671 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.549219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.131:35893)
0806 04:39:58.549529 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.549598 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225}
I20260806 04:39:58.550048 7969 mvcc.cc:205] Tried to move back new op lower bound from 7315419949225787392 to 7315419949139681280. Current Snapshot: MvccSnapshot[applied={T|T < 7315419949225787392}]
I20260806 04:39:58.550539 7929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.550053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.129:44663)
0806 04:39:58.550402 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.550460 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276}
I20260806 04:39:58.551523 7970 mvcc.cc:205] Tried to move back new op lower bound from 7315419949225787392 to 7315419949139681280. Current Snapshot: MvccSnapshot[applied={T|T < 7315419949225787392}]
I20260806 04:39:58.555725 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.555434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.131:48227)
0806 04:39:58.555591 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.555645 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 04:39:58.556190 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.555910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43117 (local address 127.31.166.130:48743)
0806 04:39:58.556057 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.556114 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 04:39:58.558876 7147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:58.559015 7147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:39:58.562937 7147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37c687289f6c45a785cb45ca13c8f276: Sending DeleteTablet for 3 replicas of tablet 59054fa12c094bf7a0a3d2b33d2554d8
I20260806 04:39:58.563882 7469 tablet_service.cc:1514] Processing DeleteTablet for tablet 59054fa12c094bf7a0a3d2b33d2554d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:58 UTC) from {username='oryx', principal='oryx/127.31.166.189@KRBTEST.COM'} at 127.0.0.1:52740
I20260806 04:39:58.563933 7760 tablet_service.cc:1514] Processing DeleteTablet for tablet 59054fa12c094bf7a0a3d2b33d2554d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:58 UTC) from {username='oryx', principal='oryx/127.31.166.189@KRBTEST.COM'} at 127.0.0.1:56280
I20260806 04:39:58.564218 7615 tablet_service.cc:1514] Processing DeleteTablet for tablet 59054fa12c094bf7a0a3d2b33d2554d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:39:58 UTC) from {username='oryx', principal='oryx/127.31.166.189@KRBTEST.COM'} at 127.0.0.1:50420
I20260806 04:39:58.564285 7979 tablet_replica.cc:330] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de: stopping tablet replica
I20260806 04:39:58.564394 7979 raft_consensus.cc:2229] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:39:58.564595 7981 tablet_replica.cc:330] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54: stopping tablet replica
I20260806 04:39:58.564723 7981 raft_consensus.cc:2229] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:58.564719 7979 raft_consensus.cc:2258] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:58.564867 7981 raft_consensus.cc:2258] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:58.565001 7980 tablet_replica.cc:330] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38: stopping tablet replica
I20260806 04:39:58.565122 7980 raft_consensus.cc:2229] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:39:58.565258 7980 raft_consensus.cc:2258] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:39:58.565702 7981 ts_tablet_manager.cc:1905] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:58.565994 7980 ts_tablet_manager.cc:1905] T 59054fa12c094bf7a0a3d2b33d2554d8 P 0c2609e68bce4b6c8edc78b221bd9f38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:58.566129 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 7305
I20260806 04:39:58.569029 7979 ts_tablet_manager.cc:1905] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:39:58.569664 7981 ts_tablet_manager.cc:1918] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:39:58.569736 7981 log.cc:1196] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/59054fa12c094bf7a0a3d2b33d2554d8
I20260806 04:39:58.570035 7981 ts_tablet_manager.cc:1939] T 59054fa12c094bf7a0a3d2b33d2554d8 P ee19bfd1525e40929731e010edb80f54: Deleting consensus metadata
I20260806 04:39:58.571229 7133 catalog_manager.cc:4841] TS ee19bfd1525e40929731e010edb80f54 (127.31.166.130:35665): tablet 59054fa12c094bf7a0a3d2b33d2554d8 (table test-table [id=b92ed1b9d3f842e39d03eec45be3300f]) successfully deleted
I20260806 04:39:58.571807 7979 ts_tablet_manager.cc:1918] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:39:58.571883 7979 log.cc:1196] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/59054fa12c094bf7a0a3d2b33d2554d8
I20260806 04:39:58.572160 7979 ts_tablet_manager.cc:1939] T 59054fa12c094bf7a0a3d2b33d2554d8 P 595843906abc4b7386232904fb4280de: Deleting consensus metadata
I20260806 04:39:58.572850 7130 catalog_manager.cc:4841] TS 595843906abc4b7386232904fb4280de (127.31.166.131:37535): tablet 59054fa12c094bf7a0a3d2b33d2554d8 (table test-table [id=b92ed1b9d3f842e39d03eec45be3300f]) successfully deleted
I20260806 04:39:58.577543 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 7546
I20260806 04:39:58.578671 7950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:39:58.577590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:39873 (local address 127.0.0.1:52754)
0806 04:39:58.578512 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0806 04:39:58.578585 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:39873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":816}
W20260806 04:39:58.579017 7133 catalog_manager.cc:4584] TS 0c2609e68bce4b6c8edc78b221bd9f38 (127.31.166.129:39873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59054fa12c094bf7a0a3d2b33d2554d8: Network error: Client connection negotiation failed: client connection to 127.31.166.129:39873: connect: Connection refused (error 111)
I20260806 04:39:58.585950 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 7693
I20260806 04:39:58.593998 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 6996
I20260806 04:39:58.602144 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 7109
2026-08-06T04:39:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7190 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:39:58 dist-test-slave-vkjp krb5kdc[7984](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:39:58 dist-test-slave-vkjp krb5kdc[7984](info): set up 2 sockets
Aug 06 04:39:58 dist-test-slave-vkjp krb5kdc[7984](info): commencing operation
krb5kdc: starting...
W20260806 04:39:59.771471 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:39:59 dist-test-slave-vkjp krb5kdc[7984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:39:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:39:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.190.keytab.
Entry for principal oryx/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.190.keytab.
I20260806 04:39:59.900254 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:37637
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.190.keytab
--principal=oryx/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:37637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:40:00.012499 7999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:00.012797 7999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:00.012838 7999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:00.016939 7999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:40:00.017006 7999 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 04:40:00.017024 7999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:00.017042 7999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:00.017059 7999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:40:00.017076 7999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:40:00.022436 7999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:37637
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.190.keytab
--principal=oryx/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:37637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:00.023583 7999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:00.024621 7999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:00.033408 8007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:00.033422 8005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:00.033532 7999 server_base.cc:1034] running on GCE node
W20260806 04:40:00.033427 8010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:00.034376 7999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:00.035872 7999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:00.037081 7999 hybrid_clock.cc:648] HybridClock initialized: now 1785991200037055 us; error 39 us; skew 500 ppm
Aug 06 04:40:00 dist-test-slave-vkjp krb5kdc[7984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991200, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:00.040059 7999 init.cc:377] Logged in from keytab as oryx/127.31.166.190@KRBTEST.COM (short username oryx)
I20260806 04:40:00.041042 7999 webserver.cc:460] Webserver started at http://127.31.166.190:45311/ using document root <none> and password file <none>
I20260806 04:40:00.041617 7999 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:00.041713 7999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:00.041957 7999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:00.044153 7999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "6d60c3ffdadb4b4cb2bddffb12723615"
format_stamp: "Formatted at 2026-08-06 04:40:00 on dist-test-slave-vkjp"
server_key: "2519501247a514958720a104657b367f"
server_key_iv: "328084ba3127c99a3a63b5ac40eb16bd"
server_key_version: "encryptionkey@0"
I20260806 04:40:00.044754 7999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "6d60c3ffdadb4b4cb2bddffb12723615"
format_stamp: "Formatted at 2026-08-06 04:40:00 on dist-test-slave-vkjp"
server_key: "2519501247a514958720a104657b367f"
server_key_iv: "328084ba3127c99a3a63b5ac40eb16bd"
server_key_version: "encryptionkey@0"
I20260806 04:40:00.048753 7999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 04:40:00.051342 8017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:00.052613 7999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:40:00.052783 7999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "6d60c3ffdadb4b4cb2bddffb12723615"
format_stamp: "Formatted at 2026-08-06 04:40:00 on dist-test-slave-vkjp"
server_key: "2519501247a514958720a104657b367f"
server_key_iv: "328084ba3127c99a3a63b5ac40eb16bd"
server_key_version: "encryptionkey@0"
I20260806 04:40:00.052930 7999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:00.069590 7999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:00.073016 7999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:00.073231 7999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:00.081882 7999 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:37637
I20260806 04:40:00.081928 8083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:37637 every 8 connection(s)
I20260806 04:40:00.083168 7999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:40:00.086351 8084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:00.087136 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 7999
I20260806 04:40:00.087285 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:40:00.087596 32410 external_mini_cluster.cc:1428] Setting key 0f337a386d8f3ebfad0a8b2e4f511c55
I20260806 04:40:00.093400 8084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615: Bootstrap starting.
I20260806 04:40:00.096460 8084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615: Neither blocks nor log segments found. Creating new log.
Aug 06 04:40:00 dist-test-slave-vkjp krb5kdc[7984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
I20260806 04:40:00.097347 8084 log.cc:824] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615: Log is configured to *not* fsync() on all Append() calls
I20260806 04:40:00.099781 8084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615: No bootstrap required, opened a new log
I20260806 04:40:00.102845 8084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d60c3ffdadb4b4cb2bddffb12723615" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37637 } }
I20260806 04:40:00.103116 8084 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:00.103209 8084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d60c3ffdadb4b4cb2bddffb12723615, State: Initialized, Role: FOLLOWER
I20260806 04:40:00.103605 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.089007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.31.166.190:37637)
0806 04:40:00.089422 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:00.089438 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:00.089474 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:40:00.090105 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:00.090121 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:00.090453 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:00.090725 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:00.091831 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.092811 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.093979 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.094449 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.094566 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.097761 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:00.097771 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:00.097792 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:00.097830 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:00.100430 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.100938 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.100943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.100947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.101090 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.101393 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.101397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.101400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.101795 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:00.102026 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:00.102321 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 04:40:00.102635 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":132,"threads_started":1}
I20260806 04:40:00.103649 8084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d60c3ffdadb4b4cb2bddffb12723615" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37637 } }
I20260806 04:40:00.103853 8084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:40:00.103931 8084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:40:00.104058 8084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:00.105345 8084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d60c3ffdadb4b4cb2bddffb12723615" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37637 } }
I20260806 04:40:00.105701 8084 leader_election.cc:302] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d60c3ffdadb4b4cb2bddffb12723615; no voters:
I20260806 04:40:00.106086 8084 leader_election.cc:288] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:40:00.106161 8089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:40:00.106419 8089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [term 1 LEADER]: Becoming Leader. State: Replica: 6d60c3ffdadb4b4cb2bddffb12723615, State: Running, Role: LEADER
I20260806 04:40:00.106719 8089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d60c3ffdadb4b4cb2bddffb12723615" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37637 } }
I20260806 04:40:00.107465 8084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:40:00.108669 8091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d60c3ffdadb4b4cb2bddffb12723615. Latest consensus state: current_term: 1 leader_uuid: "6d60c3ffdadb4b4cb2bddffb12723615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d60c3ffdadb4b4cb2bddffb12723615" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37637 } } }
I20260806 04:40:00.108704 8090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d60c3ffdadb4b4cb2bddffb12723615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d60c3ffdadb4b4cb2bddffb12723615" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37637 } } }
I20260806 04:40:00.108815 8091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [sys.catalog]: This master's current role is: LEADER
I20260806 04:40:00.108834 8090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615 [sys.catalog]: This master's current role is: LEADER
I20260806 04:40:00.109349 8100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:40:00.112207 8100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:40:00.118139 8100 catalog_manager.cc:1347] Generated new cluster ID: b7f6f65b7cd74af2a5d8d492cae2790c
I20260806 04:40:00.118228 8100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:40:00.127022 8100 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:40:00.128115 8100 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:40:00.135022 8100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615: Generated new TSK 0
I20260806 04:40:00.135762 8100 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.129.keytab.
Entry for principal oryx/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.129.keytab.
I20260806 04:40:00.167385 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:37637
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.129.keytab
--principal=oryx/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:40:00.284520 8113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:00.284775 8113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:00.284821 8113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:00.288872 8113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:00.288936 8113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:00.288977 8113 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:40:00.289011 8113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:40:00.294811 8113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.129.keytab
--principal=oryx/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.166.190:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:00.295907 8113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:00.296871 8113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:00.306550 8119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:00.306578 8118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:00.306730 8113 server_base.cc:1034] running on GCE node
W20260806 04:40:00.306555 8121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:00.307339 8113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:00.308051 8113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:00.309250 8113 hybrid_clock.cc:648] HybridClock initialized: now 1785991200309225 us; error 32 us; skew 500 ppm
Aug 06 04:40:00 dist-test-slave-vkjp krb5kdc[7984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991200, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:00.312333 8113 init.cc:377] Logged in from keytab as oryx/127.31.166.129@KRBTEST.COM (short username oryx)
I20260806 04:40:00.313225 8113 webserver.cc:460] Webserver started at http://127.31.166.129:40413/ using document root <none> and password file <none>
I20260806 04:40:00.313807 8113 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:00.313905 8113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:00.314152 8113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:00.316282 8113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995"
format_stamp: "Formatted at 2026-08-06 04:40:00 on dist-test-slave-vkjp"
server_key: "6183c82eb10778cce81134b8e72f265b"
server_key_iv: "3ec2a26989469f87008e7fec272ffc70"
server_key_version: "encryptionkey@0"
I20260806 04:40:00.316911 8113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995"
format_stamp: "Formatted at 2026-08-06 04:40:00 on dist-test-slave-vkjp"
server_key: "6183c82eb10778cce81134b8e72f265b"
server_key_iv: "3ec2a26989469f87008e7fec272ffc70"
server_key_version: "encryptionkey@0"
I20260806 04:40:00.320747 8113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:40:00.323480 8129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:00.324780 8113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:40:00.324955 8113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995"
format_stamp: "Formatted at 2026-08-06 04:40:00 on dist-test-slave-vkjp"
server_key: "6183c82eb10778cce81134b8e72f265b"
server_key_iv: "3ec2a26989469f87008e7fec272ffc70"
server_key_version: "encryptionkey@0"
I20260806 04:40:00.325088 8113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:00.354755 8113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:00.358704 8113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:00.358955 8113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:00.360621 8113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:00.360705 8113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:00.360788 8113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:00.360843 8113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:40:00 dist-test-slave-vkjp krb5kdc[7984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991200, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
I20260806 04:40:00.372947 8113 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:43819
I20260806 04:40:00.372984 8253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:43819 every 8 connection(s)
I20260806 04:40:00.373894 8113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:40:00.375221 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 8113
I20260806 04:40:00.375392 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:40:00.375705 32410 external_mini_cluster.cc:1428] Setting key 4ba9e2049b2d52e6c23b1e92cd050c71
I20260806 04:40:00.376117 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.361603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:58089 (local address 127.31.166.190:37637)
0806 04:40:00.361733 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:00.361736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:00.362878 (+ 1142us) server_negotiation.cc:407] Connection header received
0806 04:40:00.363922 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:00.363929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:00.364010 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:00.364122 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:00.365800 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.366699 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.368797 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.369199 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.369319 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.372249 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:00.372253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:00.372255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:00.372291 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:00.373866 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.374459 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.374463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.374465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.374554 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.374968 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.374971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.374973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.375227 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:00.375336 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:00.375816 (+ 480us) server_negotiation.cc:299] Negotiation successful
0806 04:40:00.375944 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:40:00.376853 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.362198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:37637 (local address 127.31.166.129:58089)
0806 04:40:00.362715 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:00.362757 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:00.363671 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:00.364388 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:00.364398 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:00.364909 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:00.365607 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.365628 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.366859 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.366863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.368638 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.368652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.369357 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.369361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.369570 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.370224 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:00.370240 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:00.372079 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:00.374013 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:00.374018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:00.374031 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:00.374336 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:00.374663 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:00.374669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:00.374672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:00.374830 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:00.375335 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:00.375342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:00.375579 (+ 237us) client_negotiation.cc:769] Sending connection context
0806 04:40:00.375755 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 04:40:00.376066 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":150,"threads_started":1}
I20260806 04:40:00.378300 8254 heartbeater.cc:347] Connected to a master server at 127.31.166.190:37637
I20260806 04:40:00.378445 8254 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:00.379017 8254 heartbeater.cc:511] Master 127.31.166.190:37637 requested a full tablet report, sending...
I20260806 04:40:00.380779 8035 ts_manager.cc:194] Registered new tserver with Master: e87eb3fa5e0b44bca1d2a0ea3d0c1995 (127.31.166.129:43819)
I20260806 04:40:00.382130 8035 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:58089
WARNING: no policy specified for oryx/127.31.166.130@KRBTEST.COM; defaulting to no policy
I20260806 04:40:00.389024 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.380511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:34383 (local address 127.31.166.190:37637)
0806 04:40:00.380617 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:00.380622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:00.380872 (+ 250us) server_negotiation.cc:407] Connection header received
0806 04:40:00.381052 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:00.381059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:00.381122 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:00.381220 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:00.381725 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.382701 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.383821 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.384149 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.384260 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.385703 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:00.385706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:00.385707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:00.385729 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:00.387268 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.387790 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.387793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.387794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.387878 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.388205 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.388207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.388208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.388361 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:00.388438 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:00.388795 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 04:40:00.388906 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:40:00.389097 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.380445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:37637 (local address 127.31.166.129:34383)
0806 04:40:00.380786 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:00.380803 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:00.380909 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:00.381251 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:00.381253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:00.381467 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:00.381589 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.381596 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.382890 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.382896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.383692 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.383702 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.384299 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.384305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.384450 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.384984 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:00.385007 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:00.385549 (+ 542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:00.387383 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:00.387390 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:00.387393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:00.387682 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:00.387981 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:00.387987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:00.387991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:00.388104 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:00.388473 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:00.388477 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:00.388605 (+ 128us) client_negotiation.cc:769] Sending connection context
0806 04:40:00.388743 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 04:40:00.388921 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.166.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.130.keytab.
Entry for principal oryx/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.130.keytab.
I20260806 04:40:00.408368 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:37637
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.130.keytab
--principal=oryx/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:40:00.523411 8263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:00.523690 8263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:00.523738 8263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:00.527873 8263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:00.527935 8263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:00.527976 8263 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:40:00.528010 8263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:40:00.533361 8263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.130.keytab
--principal=oryx/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.166.190:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:00.534446 8263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:00.535444 8263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:00.545730 8272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:00.545719 8269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:00.545804 8263 server_base.cc:1034] running on GCE node
W20260806 04:40:00.545740 8270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:00.546538 8263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:00.547328 8263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:00.548563 8263 hybrid_clock.cc:648] HybridClock initialized: now 1785991200548540 us; error 26 us; skew 500 ppm
Aug 06 04:40:00 dist-test-slave-vkjp krb5kdc[7984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991200, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:00.551687 8263 init.cc:377] Logged in from keytab as oryx/127.31.166.130@KRBTEST.COM (short username oryx)
I20260806 04:40:00.552662 8263 webserver.cc:460] Webserver started at http://127.31.166.130:33947/ using document root <none> and password file <none>
I20260806 04:40:00.553282 8263 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:00.553352 8263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:00.553586 8263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:00.556176 8263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "3674247fb9b34a4bb13e38e6a33fde95"
format_stamp: "Formatted at 2026-08-06 04:40:00 on dist-test-slave-vkjp"
server_key: "5da3c6e00e5f75f9e402a9449f9dfd5c"
server_key_iv: "dc0262409b94a8913000fac4bfa9b655"
server_key_version: "encryptionkey@0"
I20260806 04:40:00.556878 8263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "3674247fb9b34a4bb13e38e6a33fde95"
format_stamp: "Formatted at 2026-08-06 04:40:00 on dist-test-slave-vkjp"
server_key: "5da3c6e00e5f75f9e402a9449f9dfd5c"
server_key_iv: "dc0262409b94a8913000fac4bfa9b655"
server_key_version: "encryptionkey@0"
I20260806 04:40:00.561241 8263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 04:40:00.564024 8279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:00.565297 8263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:40:00.565434 8263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "3674247fb9b34a4bb13e38e6a33fde95"
format_stamp: "Formatted at 2026-08-06 04:40:00 on dist-test-slave-vkjp"
server_key: "5da3c6e00e5f75f9e402a9449f9dfd5c"
server_key_iv: "dc0262409b94a8913000fac4bfa9b655"
server_key_version: "encryptionkey@0"
I20260806 04:40:00.565565 8263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:00.583717 8263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:00.587117 8263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:00.587332 8263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:00.588845 8263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:00.588930 8263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:00.589000 8263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:00.589046 8263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:40:00 dist-test-slave-vkjp krb5kdc[7984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991200, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
I20260806 04:40:00.600607 8263 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:35963
I20260806 04:40:00.600693 8411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:35963 every 8 connection(s)
I20260806 04:40:00.601845 8263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:40:00.604499 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.590040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:43281 (local address 127.31.166.190:37637)
0806 04:40:00.590243 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:00.590247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:00.591366 (+ 1119us) server_negotiation.cc:407] Connection header received
0806 04:40:00.592547 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:00.592551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:00.592616 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:00.592737 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:00.594448 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.595435 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.596990 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.597305 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.597450 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.600435 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:00.600439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:00.600443 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:00.600477 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:00.602162 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.602746 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.602750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.602754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.602862 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.603362 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.603365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.603367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.603588 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:00.603674 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:00.604269 (+ 595us) server_negotiation.cc:299] Negotiation successful
0806 04:40:00.604375 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 04:40:00.605461 8290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.590624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:37637 (local address 127.31.166.130:43281)
0806 04:40:00.591202 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:00.591250 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:00.592316 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:00.592960 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:00.592970 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:00.593479 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:00.594289 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.594311 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.595572 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.595575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.596841 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.596861 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.597494 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.597499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.597695 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.598384 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:00.598405 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:00.600207 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:00.602313 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:00.602320 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:00.602335 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:00.602644 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:00.602993 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:00.602996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:00.602998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:00.603223 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:00.603717 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:00.603722 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:00.604013 (+ 291us) client_negotiation.cc:769] Sending connection context
0806 04:40:00.604208 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 04:40:00.604519 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":128,"threads_started":1}
I20260806 04:40:00.605659 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 8263
I20260806 04:40:00.605806 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:40:00.606200 32410 external_mini_cluster.cc:1428] Setting key 7789ecca24755fd3ce28836eb5b7d776
I20260806 04:40:00.606806 8412 heartbeater.cc:347] Connected to a master server at 127.31.166.190:37637
I20260806 04:40:00.606972 8412 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:00.607635 8412 heartbeater.cc:511] Master 127.31.166.190:37637 requested a full tablet report, sending...
I20260806 04:40:00.609038 8035 ts_manager.cc:194] Registered new tserver with Master: 3674247fb9b34a4bb13e38e6a33fde95 (127.31.166.130:35963)
I20260806 04:40:00.609920 8035 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:43281
WARNING: no policy specified for oryx/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:40:00.618695 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.609878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:33073 (local address 127.31.166.190:37637)
0806 04:40:00.610014 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:00.610020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:00.611373 (+ 1353us) server_negotiation.cc:407] Connection header received
0806 04:40:00.611593 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:00.611598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:00.611650 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:00.611704 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:00.612427 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.613092 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.613942 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.614180 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.614292 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.615521 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:00.615524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:00.615527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:00.615550 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:00.616973 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.617433 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.617437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.617439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.617525 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.617804 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.617809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.617811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.618027 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:00.618138 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:00.618447 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 04:40:00.618558 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:40:00.618700 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.609735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:37637 (local address 127.31.166.130:33073)
0806 04:40:00.611272 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:00.611285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:00.611458 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:00.611857 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:00.611861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:00.612135 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:00.612284 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.612294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.613215 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.613217 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.613813 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.613822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.614299 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.614302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.614402 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.614950 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:00.614962 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:00.615380 (+ 418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:00.617094 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:00.617099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:00.617101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:00.617324 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:00.617618 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:00.617622 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:00.617624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:00.617714 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:00.618128 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:00.618131 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:00.618229 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 04:40:00.618338 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 04:40:00.618483 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1460,"thread_start_us":92,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.166.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.131.keytab.
Entry for principal oryx/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.131.keytab.
I20260806 04:40:00.639250 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:37637
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.131.keytab
--principal=oryx/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:40:00.756330 8422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:00.756646 8422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:00.756708 8422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:00.760876 8422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:00.760950 8422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:00.761472 8422 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 04:40:00.761533 8422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:40:00.766940 8422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.131.keytab
--principal=oryx/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.166.190:37637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:00.768033 8422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:00.769146 8422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:00.779034 8430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:00.779124 8428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:00.779127 8427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:00.779307 8422 server_base.cc:1034] running on GCE node
I20260806 04:40:00.779840 8422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:00.780463 8422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:00.781657 8422 hybrid_clock.cc:648] HybridClock initialized: now 1785991200781642 us; error 26 us; skew 500 ppm
Aug 06 04:40:00 dist-test-slave-vkjp krb5kdc[7984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991200, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:00.784499 8422 init.cc:377] Logged in from keytab as oryx/127.31.166.131@KRBTEST.COM (short username oryx)
I20260806 04:40:00.785367 8422 webserver.cc:460] Webserver started at http://127.31.166.131:34363/ using document root <none> and password file <none>
I20260806 04:40:00.785925 8422 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:00.786036 8422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:00.786262 8422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:00.788328 8422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "755d861066ec45fdba645b4a4518d95b"
format_stamp: "Formatted at 2026-08-06 04:40:00 on dist-test-slave-vkjp"
server_key: "41d787bad4e950d260103691a04f0655"
server_key_iv: "e6a77e24adea46035e66cee219ceb02e"
server_key_version: "encryptionkey@0"
I20260806 04:40:00.788890 8422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "755d861066ec45fdba645b4a4518d95b"
format_stamp: "Formatted at 2026-08-06 04:40:00 on dist-test-slave-vkjp"
server_key: "41d787bad4e950d260103691a04f0655"
server_key_iv: "e6a77e24adea46035e66cee219ceb02e"
server_key_version: "encryptionkey@0"
I20260806 04:40:00.792512 8422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:40:00.794847 8438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:00.796067 8422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:40:00.796206 8422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "755d861066ec45fdba645b4a4518d95b"
format_stamp: "Formatted at 2026-08-06 04:40:00 on dist-test-slave-vkjp"
server_key: "41d787bad4e950d260103691a04f0655"
server_key_iv: "e6a77e24adea46035e66cee219ceb02e"
server_key_version: "encryptionkey@0"
I20260806 04:40:00.796320 8422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:00.808476 8422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:00.811770 8422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:00.811977 8422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:00.813418 8422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:00.813489 8422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:00.813558 8422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:00.813602 8422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 04:40:00 dist-test-slave-vkjp krb5kdc[7984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991200, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
I20260806 04:40:00.824880 8422 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:41597
I20260806 04:40:00.824936 8562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:41597 every 8 connection(s)
I20260806 04:40:00.825737 8422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:40:00.826316 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 8422
I20260806 04:40:00.826471 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:40:00.826776 32410 external_mini_cluster.cc:1428] Setting key 6bfdad90fec37af84a3a1cbb8a652c7f
I20260806 04:40:00.828431 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.814620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:37637 (local address 127.31.166.131:40167)
0806 04:40:00.815149 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:00.815192 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:00.816114 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:00.816811 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:00.816820 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:00.817222 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:00.817795 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.817809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.818915 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.818919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.819908 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.819920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.820504 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.820510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.820747 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.821582 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:00.821604 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:00.823459 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:00.825427 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:00.825433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:00.825446 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:00.825715 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:00.826083 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:00.826086 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:00.826088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:00.826261 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:00.826847 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:00.826852 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:00.827127 (+ 275us) client_negotiation.cc:769] Sending connection context
0806 04:40:00.827317 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 04:40:00.827641 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":137,"threads_started":1}
I20260806 04:40:00.829241 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.814120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:40167 (local address 127.31.166.190:37637)
0806 04:40:00.814244 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:00.814248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:00.815297 (+ 1049us) server_negotiation.cc:407] Connection header received
0806 04:40:00.816374 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:00.816381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:00.816464 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:00.816565 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:00.818060 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.818767 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.820051 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.820294 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.820431 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.823634 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:00.823638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:00.823640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:00.823664 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:00.825277 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.825853 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.825857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.825859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.825956 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.826390 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.826397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.826400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.826660 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:00.826756 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:00.829039 (+ 2283us) server_negotiation.cc:299] Negotiation successful
0806 04:40:00.829136 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:40:00.830158 8564 heartbeater.cc:347] Connected to a master server at 127.31.166.190:37637
I20260806 04:40:00.830327 8564 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:00.830969 8564 heartbeater.cc:511] Master 127.31.166.190:37637 requested a full tablet report, sending...
I20260806 04:40:00.832036 8036 ts_manager.cc:194] Registered new tserver with Master: 755d861066ec45fdba645b4a4518d95b (127.31.166.131:41597)
I20260806 04:40:00.832810 8036 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:40167
I20260806 04:40:00.839839 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.832631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:38497 (local address 127.31.166.190:37637)
0806 04:40:00.832759 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:00.832764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:00.832952 (+ 188us) server_negotiation.cc:407] Connection header received
0806 04:40:00.833097 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:00.833101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:00.833144 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:00.833228 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:00.833776 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.834269 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.835278 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.835529 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.835606 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.836605 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:00.836608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:00.836610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:00.836630 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:00.838063 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.838519 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.838522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.838523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.838604 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.838932 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.838934 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.838935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.839181 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:00.839293 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:00.839584 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 04:40:00.839696 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:40:00.839852 8571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.832558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:37637 (local address 127.31.166.131:38497)
0806 04:40:00.832863 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:00.832876 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:00.832992 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:00.833265 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:00.833269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:00.833527 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:00.833655 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.833664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.834405 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.834409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.835145 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.835157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.835631 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.835634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.835706 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.836134 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:00.836146 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:00.836482 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:00.838174 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:00.838177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:00.838178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:00.838371 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:00.838747 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:00.838750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:00.838751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:00.838839 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:00.839297 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:00.839301 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:00.839397 (+ 96us) client_negotiation.cc:769] Sending connection context
0806 04:40:00.839514 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 04:40:00.839659 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":114,"threads_started":1}
I20260806 04:40:00.840585 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 04:40:00 dist-test-slave-vkjp krb5kdc[7984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 04:40:00 dist-test-slave-vkjp krb5kdc[7984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:00 dist-test-slave-vkjp krb5kdc[7984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 04:40:00.865770 8087 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:00.858470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36630 (local address 127.31.166.190:37637)
0806 04:40:00.858646 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:00.858650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:00.858664 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:40:00.858706 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:00.858708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:00.858750 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:00.858884 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:00.859806 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.860394 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.861373 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.861619 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.861746 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.865553 (+ 3807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36630: BlockingRecv error: recv got EOF from 127.0.0.1:36630 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 04:40:00 dist-test-slave-vkjp krb5kdc[7984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991200, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
I20260806 04:40:00.875715 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.867208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.31.166.190:37637)
0806 04:40:00.867368 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:00.867373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:00.867458 (+ 85us) server_negotiation.cc:407] Connection header received
0806 04:40:00.867558 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:00.867560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:00.867597 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:00.867664 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:00.868329 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.868809 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.869736 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.869958 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.870033 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.872522 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:00.872526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:00.872528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:00.872550 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:00.873986 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.874452 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.874455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.874456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.874527 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:00.874775 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:00.874777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:00.874778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:00.874957 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:00.875131 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:00.875420 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 04:40:00.875543 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 04:40:00.878926 8035 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:40:00.881965 8035 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:40:00.897265 8588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.890949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:43819 (local address 127.0.0.1:37384)
0806 04:40:00.891402 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:00.891462 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:00.891605 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:00.892495 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:00.892502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:00.892531 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:40:00.892714 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.892722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.894170 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.894174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.895474 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.895487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.896663 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.896667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.896809 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.896854 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 04:40:00.896971 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 04:40:00.897092 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":3,"thread_start_us":124,"threads_started":1}
I20260806 04:40:00.897259 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.891149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37384 (local address 127.31.166.129:43819)
0806 04:40:00.891914 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:00.891922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:00.891950 (+ 28us) server_negotiation.cc:407] Connection header received
0806 04:40:00.892035 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:00.892042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:00.892291 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:00.892900 (+ 609us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:40:00.893081 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.894052 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.895628 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.896519 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.896685 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.896858 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:00.897010 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 04:40:00.897108 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":95,"threads_started":1}
I20260806 04:40:00.898034 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.891612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:35963 (local address 127.0.0.1:52850)
0806 04:40:00.892058 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:00.892074 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:00.892168 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:00.892978 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:00.892982 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:00.893006 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:40:00.893157 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.893166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.894889 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.894894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.896267 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.896279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.897620 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.897623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.897702 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.897719 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 04:40:00.897827 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 04:40:00.897898 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":63,"thread_start_us":109,"threads_started":1}
I20260806 04:40:00.898111 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.892129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52850 (local address 127.31.166.130:35963)
0806 04:40:00.892492 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:00.892505 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:00.892534 (+ 29us) server_negotiation.cc:407] Connection header received
0806 04:40:00.892609 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:00.892614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:00.892814 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:00.893343 (+ 529us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:40:00.893474 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.894746 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.896413 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.897299 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.897456 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.897564 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:00.897857 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 04:40:00.897950 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":135,"threads_started":1}
I20260806 04:40:00.898614 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.891755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:41597 (local address 127.0.0.1:35852)
0806 04:40:00.892896 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:00.892910 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:00.892989 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:00.893770 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:00.893773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:00.893794 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:40:00.893896 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.893902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.895365 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.895368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.896466 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:00.896477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.898065 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.898069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:00.898283 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.898309 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:40:00.898388 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 04:40:00.898458 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":27,"thread_start_us":121,"threads_started":1}
I20260806 04:40:00.899055 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:00.892232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.31.166.131:41597)
0806 04:40:00.893267 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:00.893281 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:00.893305 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:40:00.893400 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:00.893405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:00.893592 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:00.893753 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:40:00.894021 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.895213 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.896614 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:00.897717 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:00.897861 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:00.897995 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:00.898793 (+ 798us) server_negotiation.cc:299] Negotiation successful
0806 04:40:00.898897 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":132,"threads_started":1}
I20260806 04:40:00.900853 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 0183e42b9afb4b8c9203c5eab4134e2e (DEFAULT_TABLE table=test-table [id=54cb4581259649718cbe9b3726909d49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:00.901902 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0183e42b9afb4b8c9203c5eab4134e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:00.903504 8341 tablet_service.cc:1467] Processing CreateTablet for tablet 0183e42b9afb4b8c9203c5eab4134e2e (DEFAULT_TABLE table=test-table [id=54cb4581259649718cbe9b3726909d49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:00.904472 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0183e42b9afb4b8c9203c5eab4134e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:00.904939 8495 tablet_service.cc:1467] Processing CreateTablet for tablet 0183e42b9afb4b8c9203c5eab4134e2e (DEFAULT_TABLE table=test-table [id=54cb4581259649718cbe9b3726909d49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:00.905779 8495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0183e42b9afb4b8c9203c5eab4134e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:00.913547 8595 tablet_bootstrap.cc:492] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b: Bootstrap starting.
I20260806 04:40:00.912925 8594 tablet_bootstrap.cc:492] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95: Bootstrap starting.
I20260806 04:40:00.914403 8596 tablet_bootstrap.cc:492] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Bootstrap starting.
I20260806 04:40:00.916119 8595 tablet_bootstrap.cc:654] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:00.916201 8594 tablet_bootstrap.cc:654] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:00.916250 8596 tablet_bootstrap.cc:654] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:00.916980 8596 log.cc:824] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Log is configured to *not* fsync() on all Append() calls
I20260806 04:40:00.917065 8594 log.cc:824] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95: Log is configured to *not* fsync() on all Append() calls
I20260806 04:40:00.917065 8595 log.cc:824] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b: Log is configured to *not* fsync() on all Append() calls
I20260806 04:40:00.919000 8596 tablet_bootstrap.cc:492] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995: No bootstrap required, opened a new log
I20260806 04:40:00.919183 8594 tablet_bootstrap.cc:492] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95: No bootstrap required, opened a new log
I20260806 04:40:00.919183 8595 tablet_bootstrap.cc:492] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b: No bootstrap required, opened a new log
I20260806 04:40:00.919314 8596 ts_tablet_manager.cc:1397] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260806 04:40:00.919386 8594 ts_tablet_manager.cc:1397] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:40:00.919386 8595 ts_tablet_manager.cc:1397] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:40:00.923413 8596 raft_consensus.cc:348] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } }
I20260806 04:40:00.923678 8596 raft_consensus.cc:374] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:00.923738 8596 raft_consensus.cc:729] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87eb3fa5e0b44bca1d2a0ea3d0c1995, State: Initialized, Role: FOLLOWER
I20260806 04:40:00.923729 8595 raft_consensus.cc:348] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } }
I20260806 04:40:00.923729 8594 raft_consensus.cc:348] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } }
I20260806 04:40:00.923995 8595 raft_consensus.cc:374] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:00.923995 8594 raft_consensus.cc:374] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:00.924065 8594 raft_consensus.cc:729] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3674247fb9b34a4bb13e38e6a33fde95, State: Initialized, Role: FOLLOWER
I20260806 04:40:00.924065 8595 raft_consensus.cc:729] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d861066ec45fdba645b4a4518d95b, State: Initialized, Role: FOLLOWER
I20260806 04:40:00.924237 8596 consensus_queue.cc:260] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } }
I20260806 04:40:00.924523 8595 consensus_queue.cc:260] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } }
I20260806 04:40:00.924523 8594 consensus_queue.cc:260] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } }
I20260806 04:40:00.925083 8254 heartbeater.cc:503] Master 127.31.166.190:37637 was elected leader, sending a full tablet report...
I20260806 04:40:00.925406 8596 ts_tablet_manager.cc:1428] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260806 04:40:00.925558 8595 ts_tablet_manager.cc:1428] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:40:00.925558 8594 ts_tablet_manager.cc:1428] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:40:00.925916 8564 heartbeater.cc:503] Master 127.31.166.190:37637 was elected leader, sending a full tablet report...
I20260806 04:40:00.925916 8412 heartbeater.cc:503] Master 127.31.166.190:37637 was elected leader, sending a full tablet report...
W20260806 04:40:01.077718 8565 tablet.cc:2369] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:40:01.103173 8413 tablet.cc:2369] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:40:01.122606 8602 raft_consensus.cc:482] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:40:01.122881 8602 raft_consensus.cc:504] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } }
I20260806 04:40:01.125051 8602 leader_election.cc:288] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e87eb3fa5e0b44bca1d2a0ea3d0c1995 (127.31.166.129:43819), 3674247fb9b34a4bb13e38e6a33fde95 (127.31.166.130:35963)
W20260806 04:40:01.125413 8255 tablet.cc:2369] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:40:01.129354 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.124997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:45297 (local address 127.31.166.130:35963)
0806 04:40:01.125157 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.125162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.125178 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:40:01.125362 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.125368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.125432 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.125533 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:40:01.125860 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.126605 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.128054 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.128608 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.128753 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.128835 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:01.129135 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 04:40:01.129217 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:40:01.129375 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.124899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:35963 (local address 127.31.166.131:45297)
0806 04:40:01.125050 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:01.125070 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:01.125195 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:01.125570 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:01.125573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:01.125597 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:40:01.125731 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.125739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.126734 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.126737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.127910 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.127921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.128750 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.128754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.128965 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.128985 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 04:40:01.129114 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 04:40:01.129197 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":58}
I20260806 04:40:01.130040 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0183e42b9afb4b8c9203c5eab4134e2e" candidate_uuid: "755d861066ec45fdba645b4a4518d95b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3674247fb9b34a4bb13e38e6a33fde95" is_pre_election: true
I20260806 04:40:01.130429 8362 raft_consensus.cc:2454] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d861066ec45fdba645b4a4518d95b in term 0.
I20260806 04:40:01.130895 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.125122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:56147 (local address 127.31.166.129:43819)
0806 04:40:01.125298 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.125303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.126147 (+ 844us) server_negotiation.cc:407] Connection header received
0806 04:40:01.126265 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.126268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.126317 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.126414 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:40:01.127423 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.127963 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.129300 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.130061 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.130235 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.130296 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:01.130700 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 04:40:01.130797 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:40:01.131049 8439 leader_election.cc:302] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3674247fb9b34a4bb13e38e6a33fde95, 755d861066ec45fdba645b4a4518d95b; no voters:
I20260806 04:40:01.131371 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.124899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:43819 (local address 127.31.166.131:56147)
0806 04:40:01.126036 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:01.126051 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:01.126179 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:01.126525 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:01.126528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:01.126546 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:40:01.127258 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.127271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.128088 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.128093 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.129165 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.129176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.130421 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.130426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.130538 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.130563 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 04:40:01.131055 (+ 492us) client_negotiation.cc:240] Negotiation successful
0806 04:40:01.131171 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"mutex_wait_us":485,"thread_start_us":119,"threads_started":1}
I20260806 04:40:01.131441 8602 raft_consensus.cc:2781] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:40:01.131551 8602 raft_consensus.cc:482] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:40:01.131625 8602 raft_consensus.cc:3037] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:01.132019 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0183e42b9afb4b8c9203c5eab4134e2e" candidate_uuid: "755d861066ec45fdba645b4a4518d95b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" is_pre_election: true
I20260806 04:40:01.132311 8205 raft_consensus.cc:2454] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 755d861066ec45fdba645b4a4518d95b in term 0.
I20260806 04:40:01.133090 8602 raft_consensus.cc:504] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } }
I20260806 04:40:01.133471 8602 leader_election.cc:288] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [CANDIDATE]: Term 1 election: Requested vote from peers e87eb3fa5e0b44bca1d2a0ea3d0c1995 (127.31.166.129:43819), 3674247fb9b34a4bb13e38e6a33fde95 (127.31.166.130:35963)
I20260806 04:40:01.133811 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0183e42b9afb4b8c9203c5eab4134e2e" candidate_uuid: "755d861066ec45fdba645b4a4518d95b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995"
I20260806 04:40:01.133931 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0183e42b9afb4b8c9203c5eab4134e2e" candidate_uuid: "755d861066ec45fdba645b4a4518d95b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3674247fb9b34a4bb13e38e6a33fde95"
I20260806 04:40:01.133970 8205 raft_consensus.cc:3037] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:01.134087 8362 raft_consensus.cc:3037] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:01.135440 8205 raft_consensus.cc:2454] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d861066ec45fdba645b4a4518d95b in term 1.
I20260806 04:40:01.135501 8362 raft_consensus.cc:2454] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 755d861066ec45fdba645b4a4518d95b in term 1.
I20260806 04:40:01.135809 8441 leader_election.cc:302] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d861066ec45fdba645b4a4518d95b, e87eb3fa5e0b44bca1d2a0ea3d0c1995; no voters:
I20260806 04:40:01.136018 8602 raft_consensus.cc:2781] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:40:01.136343 8602 raft_consensus.cc:686] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [term 1 LEADER]: Becoming Leader. State: Replica: 755d861066ec45fdba645b4a4518d95b, State: Running, Role: LEADER
I20260806 04:40:01.136757 8602 consensus_queue.cc:237] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } }
I20260806 04:40:01.140472 8034 catalog_manager.cc:5495] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b reported cstate change: term changed from 0 to 1, leader changed from <none> to 755d861066ec45fdba645b4a4518d95b (127.31.166.131). New cstate: current_term: 1 leader_uuid: "755d861066ec45fdba645b4a4518d95b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } health_report { overall_health: UNKNOWN } } }
I20260806 04:40:01.193959 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.190005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35856 (local address 127.31.166.131:41597)
0806 04:40:01.190161 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.190164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.190181 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:40:01.190309 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.190312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.190381 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.190494 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:40:01.190801 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.191456 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.192360 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.192623 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.192714 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.193024 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:40:01.193532 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:40:01.193631 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:01.193774 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 04:40:01.193837 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:40:01.200469 8205 raft_consensus.cc:1261] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Refusing update from remote peer 755d861066ec45fdba645b4a4518d95b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:40:01.200469 8362 raft_consensus.cc:1261] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Refusing update from remote peer 755d861066ec45fdba645b4a4518d95b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:40:01.201346 8605 consensus_queue.cc:1035] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:40:01.201572 8602 consensus_queue.cc:1035] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:40:01.208827 8608 mvcc.cc:205] Tried to move back new op lower bound from 7315419960108781568 to 7315419959859212288. Current Snapshot: MvccSnapshot[applied={T|T < 7315419960108781568}]
I20260806 04:40:01.209535 8609 mvcc.cc:205] Tried to move back new op lower bound from 7315419960108781568 to 7315419959859212288. Current Snapshot: MvccSnapshot[applied={T|T < 7315419960108781568}]
I20260806 04:40:01.219281 8036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:40:01.219466 8036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:40:01.222306 8036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615: Sending DeleteTablet for 3 replicas of tablet 0183e42b9afb4b8c9203c5eab4134e2e
I20260806 04:40:01.223316 8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 0183e42b9afb4b8c9203c5eab4134e2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:01 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:52850
I20260806 04:40:01.223450 8495 tablet_service.cc:1514] Processing DeleteTablet for tablet 0183e42b9afb4b8c9203c5eab4134e2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:01 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:35852
I20260806 04:40:01.223459 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 0183e42b9afb4b8c9203c5eab4134e2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:01 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37384
I20260806 04:40:01.223953 8620 tablet_replica.cc:330] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995: stopping tablet replica
I20260806 04:40:01.224128 8622 tablet_replica.cc:330] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b: stopping tablet replica
I20260806 04:40:01.224354 8620 raft_consensus.cc:2229] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:40:01.224524 8622 raft_consensus.cc:2229] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:40:01.224697 8620 raft_consensus.cc:2258] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:40:01.224607 8036 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:40:01.225041 8036 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:40:01.225242 8618 tablet_replica.cc:330] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95: stopping tablet replica
I20260806 04:40:01.225244 8622 raft_consensus.cc:2258] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:40:01.225564 8618 raft_consensus.cc:2229] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:40:01.225880 8618 raft_consensus.cc:2258] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:40:01.226938 8620 ts_tablet_manager.cc:1905] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:40:01.227335 8618 ts_tablet_manager.cc:1905] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:40:01.230338 8620 ts_tablet_manager.cc:1918] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:40:01.230562 8620 log.cc:1196] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/wal/wals/0183e42b9afb4b8c9203c5eab4134e2e
I20260806 04:40:01.230744 8618 ts_tablet_manager.cc:1918] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:40:01.230865 8618 log.cc:1196] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/0183e42b9afb4b8c9203c5eab4134e2e
I20260806 04:40:01.231266 8618 ts_tablet_manager.cc:1939] T 0183e42b9afb4b8c9203c5eab4134e2e P 3674247fb9b34a4bb13e38e6a33fde95: Deleting consensus metadata
I20260806 04:40:01.231622 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 6f914016c3114401a6c13ff48614b83c (DEFAULT_TABLE table=test-table [id=b31d7d27ed6e4a1684b82266f3139eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:01.231671 8341 tablet_service.cc:1467] Processing CreateTablet for tablet 6f914016c3114401a6c13ff48614b83c (DEFAULT_TABLE table=test-table [id=b31d7d27ed6e4a1684b82266f3139eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:01.231912 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f914016c3114401a6c13ff48614b83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:01.231950 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f914016c3114401a6c13ff48614b83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:01.232846 8620 ts_tablet_manager.cc:1939] T 0183e42b9afb4b8c9203c5eab4134e2e P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Deleting consensus metadata
I20260806 04:40:01.234364 8018 catalog_manager.cc:4841] TS 3674247fb9b34a4bb13e38e6a33fde95 (127.31.166.130:35963): tablet 0183e42b9afb4b8c9203c5eab4134e2e (table test-table [id=54cb4581259649718cbe9b3726909d49]) successfully deleted
I20260806 04:40:01.234529 8594 tablet_bootstrap.cc:492] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95: Bootstrap starting.
I20260806 04:40:01.234547 8596 tablet_bootstrap.cc:492] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Bootstrap starting.
I20260806 04:40:01.235600 8594 tablet_bootstrap.cc:654] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:01.235615 8596 tablet_bootstrap.cc:654] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:01.236394 8622 ts_tablet_manager.cc:1905] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:40:01.236727 8020 catalog_manager.cc:4841] TS e87eb3fa5e0b44bca1d2a0ea3d0c1995 (127.31.166.129:43819): tablet 0183e42b9afb4b8c9203c5eab4134e2e (table test-table [id=54cb4581259649718cbe9b3726909d49]) successfully deleted
I20260806 04:40:01.237190 8495 tablet_service.cc:1467] Processing CreateTablet for tablet 6f914016c3114401a6c13ff48614b83c (DEFAULT_TABLE table=test-table [id=b31d7d27ed6e4a1684b82266f3139eb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:01.237490 8495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f914016c3114401a6c13ff48614b83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:01.239681 8622 ts_tablet_manager.cc:1918] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:40:01.239804 8622 log.cc:1196] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/0183e42b9afb4b8c9203c5eab4134e2e
I20260806 04:40:01.240067 8595 tablet_bootstrap.cc:492] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b: Bootstrap starting.
I20260806 04:40:01.240181 8622 ts_tablet_manager.cc:1939] T 0183e42b9afb4b8c9203c5eab4134e2e P 755d861066ec45fdba645b4a4518d95b: Deleting consensus metadata
I20260806 04:40:01.241112 8595 tablet_bootstrap.cc:654] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:01.241430 8020 catalog_manager.cc:4841] TS 755d861066ec45fdba645b4a4518d95b (127.31.166.131:41597): tablet 0183e42b9afb4b8c9203c5eab4134e2e (table test-table [id=54cb4581259649718cbe9b3726909d49]) successfully deleted
I20260806 04:40:01.242460 8594 tablet_bootstrap.cc:492] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95: No bootstrap required, opened a new log
I20260806 04:40:01.242481 8596 tablet_bootstrap.cc:492] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995: No bootstrap required, opened a new log
I20260806 04:40:01.242594 8594 ts_tablet_manager.cc:1397] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260806 04:40:01.242610 8596 ts_tablet_manager.cc:1397] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260806 04:40:01.243121 8596 raft_consensus.cc:348] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.243249 8596 raft_consensus.cc:374] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:01.243296 8596 raft_consensus.cc:729] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87eb3fa5e0b44bca1d2a0ea3d0c1995, State: Initialized, Role: FOLLOWER
I20260806 04:40:01.243218 8594 raft_consensus.cc:348] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.243351 8595 tablet_bootstrap.cc:492] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b: No bootstrap required, opened a new log
I20260806 04:40:01.243341 8594 raft_consensus.cc:374] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:01.243388 8594 raft_consensus.cc:729] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3674247fb9b34a4bb13e38e6a33fde95, State: Initialized, Role: FOLLOWER
I20260806 04:40:01.243433 8595 ts_tablet_manager.cc:1397] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:40:01.243422 8596 consensus_queue.cc:260] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.243544 8594 consensus_queue.cc:260] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.243685 8596 ts_tablet_manager.cc:1428] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:40:01.243846 8594 ts_tablet_manager.cc:1428] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:40:01.243929 8595 raft_consensus.cc:348] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.244046 8595 raft_consensus.cc:374] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:01.244092 8595 raft_consensus.cc:729] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d861066ec45fdba645b4a4518d95b, State: Initialized, Role: FOLLOWER
I20260806 04:40:01.244216 8595 consensus_queue.cc:260] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.244534 8595 ts_tablet_manager.cc:1428] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 04:40:01.447762 8601 raft_consensus.cc:482] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:40:01.447983 8601 raft_consensus.cc:504] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.449191 8601 leader_election.cc:288] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e87eb3fa5e0b44bca1d2a0ea3d0c1995 (127.31.166.129:43819), 755d861066ec45fdba645b4a4518d95b (127.31.166.131:41597)
I20260806 04:40:01.453634 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.449636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:42613 (local address 127.31.166.129:43819)
0806 04:40:01.449846 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.449854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.449965 (+ 111us) server_negotiation.cc:407] Connection header received
0806 04:40:01.450152 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.450155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.450214 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.450376 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:40:01.450689 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.451340 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.452564 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.453001 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.453153 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.453219 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:01.453424 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 04:40:01.453503 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 04:40:01.453665 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.449540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:43819 (local address 127.31.166.130:42613)
0806 04:40:01.449822 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:01.449838 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:01.450009 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:01.450395 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:01.450398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:01.450420 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:40:01.450564 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.450571 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.451461 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.451466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.452427 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.452436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.453153 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.453157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.453268 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.453305 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 04:40:01.453404 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 04:40:01.453479 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260806 04:40:01.453765 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.450099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:44581 (local address 127.31.166.131:41597)
0806 04:40:01.450288 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.450295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.450428 (+ 133us) server_negotiation.cc:407] Connection header received
0806 04:40:01.450620 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.450623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.450681 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.450773 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:40:01.451112 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.451637 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.452722 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.453139 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.453289 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.453354 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:01.453533 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 04:40:01.453620 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:40:01.454037 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.449972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:41597 (local address 127.31.166.130:44581)
0806 04:40:01.450314 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:01.450330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:01.450487 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:01.450784 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:01.450788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:01.450806 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:40:01.450952 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.450959 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.451769 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.451774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.452579 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.452588 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.453272 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.453276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.453384 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.453410 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:40:01.453516 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 04:40:01.453588 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":25,"spinlock_wait_cycles":1280,"thread_start_us":143,"threads_started":1}
I20260806 04:40:01.454173 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f914016c3114401a6c13ff48614b83c" candidate_uuid: "3674247fb9b34a4bb13e38e6a33fde95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" is_pre_election: true
I20260806 04:40:01.454370 8205 raft_consensus.cc:2454] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3674247fb9b34a4bb13e38e6a33fde95 in term 0.
I20260806 04:40:01.454602 8515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f914016c3114401a6c13ff48614b83c" candidate_uuid: "3674247fb9b34a4bb13e38e6a33fde95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d861066ec45fdba645b4a4518d95b" is_pre_election: true
I20260806 04:40:01.454763 8284 leader_election.cc:302] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3674247fb9b34a4bb13e38e6a33fde95, e87eb3fa5e0b44bca1d2a0ea3d0c1995; no voters:
I20260806 04:40:01.454866 8515 raft_consensus.cc:2454] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3674247fb9b34a4bb13e38e6a33fde95 in term 0.
I20260806 04:40:01.455058 8601 raft_consensus.cc:2781] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:40:01.455173 8601 raft_consensus.cc:482] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:40:01.455221 8601 raft_consensus.cc:3037] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:01.456530 8601 raft_consensus.cc:504] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.456897 8601 leader_election.cc:288] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [CANDIDATE]: Term 1 election: Requested vote from peers e87eb3fa5e0b44bca1d2a0ea3d0c1995 (127.31.166.129:43819), 755d861066ec45fdba645b4a4518d95b (127.31.166.131:41597)
I20260806 04:40:01.457293 8515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f914016c3114401a6c13ff48614b83c" candidate_uuid: "3674247fb9b34a4bb13e38e6a33fde95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d861066ec45fdba645b4a4518d95b"
I20260806 04:40:01.457317 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f914016c3114401a6c13ff48614b83c" candidate_uuid: "3674247fb9b34a4bb13e38e6a33fde95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995"
I20260806 04:40:01.457441 8515 raft_consensus.cc:3037] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:01.457474 8205 raft_consensus.cc:3037] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:01.458400 8205 raft_consensus.cc:2454] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3674247fb9b34a4bb13e38e6a33fde95 in term 1.
I20260806 04:40:01.458582 8515 raft_consensus.cc:2454] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3674247fb9b34a4bb13e38e6a33fde95 in term 1.
I20260806 04:40:01.458740 8284 leader_election.cc:302] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3674247fb9b34a4bb13e38e6a33fde95, e87eb3fa5e0b44bca1d2a0ea3d0c1995; no voters:
I20260806 04:40:01.458928 8601 raft_consensus.cc:2781] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:40:01.459195 8601 raft_consensus.cc:686] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 LEADER]: Becoming Leader. State: Replica: 3674247fb9b34a4bb13e38e6a33fde95, State: Running, Role: LEADER
I20260806 04:40:01.459511 8601 consensus_queue.cc:237] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.462050 8034 catalog_manager.cc:5495] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3674247fb9b34a4bb13e38e6a33fde95 (127.31.166.130). New cstate: current_term: 1 leader_uuid: "3674247fb9b34a4bb13e38e6a33fde95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } health_report { overall_health: UNKNOWN } } }
Aug 06 04:40:01 dist-test-slave-vkjp krb5kdc[7984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991200, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.190@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
I20260806 04:40:01.514345 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.504936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.31.166.190:37637)
0806 04:40:01.505212 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.505220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.505355 (+ 135us) server_negotiation.cc:407] Connection header received
0806 04:40:01.505454 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.505456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.505498 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.505602 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:01.506390 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.506940 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.508019 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.508233 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.508312 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.510757 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:01.510760 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:01.510762 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:01.510783 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:01.512463 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:01.512954 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:01.512957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:01.512958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:01.513036 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:01.513371 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:01.513374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:01.513377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:01.513563 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:01.513642 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:01.513998 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 04:40:01.514149 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":190,"thread_start_us":95,"threads_started":1}
I20260806 04:40:01.514348 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.504816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:37637 (local address 127.0.0.1:36648)
0806 04:40:01.505206 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:01.505220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:01.505371 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:01.505686 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:01.505688 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:01.506105 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:01.506212 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.506219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.507073 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.507097 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.507886 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.507896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.508358 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.508362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.508467 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.509037 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:01.509051 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:01.510626 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:01.512571 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:01.512580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:01.512585 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:01.512861 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:01.513149 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:01.513153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:01.513155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:01.513260 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:01.513691 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:01.513698 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:01.513826 (+ 128us) client_negotiation.cc:769] Sending connection context
0806 04:40:01.513944 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 04:40:01.514083 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":142,"threads_started":1}
I20260806 04:40:01.518833 8034 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:36648:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 04:40:01.519629 8034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:40:01.525199 8341 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4ca5b8ca50431ba46071fac59d8880 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0aaa24403c904f828ee81712428cc614]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:40:01.525177 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4ca5b8ca50431ba46071fac59d8880 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0aaa24403c904f828ee81712428cc614]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:40:01.525599 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4ca5b8ca50431ba46071fac59d8880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:01.525610 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4ca5b8ca50431ba46071fac59d8880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:01.527761 8495 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4ca5b8ca50431ba46071fac59d8880 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0aaa24403c904f828ee81712428cc614]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 04:40:01.528184 8495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4ca5b8ca50431ba46071fac59d8880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:01.528355 8596 tablet_bootstrap.cc:492] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Bootstrap starting.
I20260806 04:40:01.528364 8594 tablet_bootstrap.cc:492] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95: Bootstrap starting.
I20260806 04:40:01.529434 8596 tablet_bootstrap.cc:654] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:01.529517 8594 tablet_bootstrap.cc:654] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:01.531198 8594 tablet_bootstrap.cc:492] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95: No bootstrap required, opened a new log
I20260806 04:40:01.531320 8594 ts_tablet_manager.cc:1397] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:40:01.531414 8596 tablet_bootstrap.cc:492] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: No bootstrap required, opened a new log
I20260806 04:40:01.531435 8595 tablet_bootstrap.cc:492] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b: Bootstrap starting.
I20260806 04:40:01.531507 8596 ts_tablet_manager.cc:1397] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:40:01.531824 8594 raft_consensus.cc:348] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.531960 8594 raft_consensus.cc:374] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:01.531996 8594 raft_consensus.cc:729] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3674247fb9b34a4bb13e38e6a33fde95, State: Initialized, Role: FOLLOWER
I20260806 04:40:01.531941 8596 raft_consensus.cc:348] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.532107 8596 raft_consensus.cc:374] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:01.532159 8596 raft_consensus.cc:729] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87eb3fa5e0b44bca1d2a0ea3d0c1995, State: Initialized, Role: FOLLOWER
I20260806 04:40:01.532138 8594 consensus_queue.cc:260] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.532394 8595 tablet_bootstrap.cc:654] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:01.532398 8601 tablet_replica.cc:439] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } }
I20260806 04:40:01.532547 8601 tablet_replica.cc:442] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:40:01.532469 8594 ts_tablet_manager.cc:1428] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 04:40:01.532910 8596 consensus_queue.cc:260] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.533329 8596 ts_tablet_manager.cc:1428] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 04:40:01.533258 8600 tablet_replica.cc:439] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } }
I20260806 04:40:01.533430 8600 tablet_replica.cc:442] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:40:01.533867 8595 tablet_bootstrap.cc:492] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b: No bootstrap required, opened a new log
I20260806 04:40:01.533957 8595 ts_tablet_manager.cc:1397] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:40:01.534389 8595 raft_consensus.cc:348] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.534520 8595 raft_consensus.cc:374] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:01.534569 8595 raft_consensus.cc:729] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d861066ec45fdba645b4a4518d95b, State: Initialized, Role: FOLLOWER
I20260806 04:40:01.534718 8595 consensus_queue.cc:260] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.535048 8595 ts_tablet_manager.cc:1428] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260806 04:40:01.534986 8615 tablet_replica.cc:439] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } }
I20260806 04:40:01.535144 8615 tablet_replica.cc:442] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:40:01.554658 8600 raft_consensus.cc:482] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:40:01.554808 8600 raft_consensus.cc:504] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.555961 8600 leader_election.cc:288] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3674247fb9b34a4bb13e38e6a33fde95 (127.31.166.130:35963), 755d861066ec45fdba645b4a4518d95b (127.31.166.131:41597)
I20260806 04:40:01.561127 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.556611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:47821 (local address 127.31.166.130:35963)
0806 04:40:01.556750 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.556755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.556772 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:40:01.556855 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.556858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.556898 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.556990 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:40:01.557351 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.558007 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.560072 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.560462 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.560562 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.560604 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:01.560900 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 04:40:01.560983 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:40:01.561146 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.556171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:35963 (local address 127.31.166.129:47821)
0806 04:40:01.556549 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:01.556568 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:01.556681 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:01.557011 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:01.557014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:01.557032 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:40:01.557215 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.557231 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.558126 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.558132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.559917 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.559929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.560620 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.560623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.560728 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.560752 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 04:40:01.560877 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 04:40:01.560960 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":142,"thread_start_us":165,"threads_started":1}
I20260806 04:40:01.561700 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.556319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:41597 (local address 127.31.166.129:33925)
0806 04:40:01.556783 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:01.556808 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:01.556929 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:01.557170 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:01.557173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:01.557189 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:40:01.557317 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.557322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.558172 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.558175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.559917 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.559929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.561200 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.561204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.561321 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.561347 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:40:01.561451 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 04:40:01.561518 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":163,"thread_start_us":111,"threads_started":1}
I20260806 04:40:01.561774 8593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.556473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:33925 (local address 127.31.166.131:41597)
0806 04:40:01.556621 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.556626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.556890 (+ 264us) server_negotiation.cc:407] Connection header received
0806 04:40:01.557018 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.557027 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.557069 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.557148 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:40:01.557426 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.558027 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.560654 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.561056 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.561200 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.561247 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:01.561491 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 04:40:01.561605 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:40:01.561848 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4ca5b8ca50431ba46071fac59d8880" candidate_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3674247fb9b34a4bb13e38e6a33fde95" is_pre_election: true
I20260806 04:40:01.562008 8362 raft_consensus.cc:2454] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87eb3fa5e0b44bca1d2a0ea3d0c1995 in term 0.
I20260806 04:40:01.562175 8515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4ca5b8ca50431ba46071fac59d8880" candidate_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d861066ec45fdba645b4a4518d95b" is_pre_election: true
I20260806 04:40:01.562393 8515 raft_consensus.cc:2454] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87eb3fa5e0b44bca1d2a0ea3d0c1995 in term 0.
I20260806 04:40:01.562352 8130 leader_election.cc:302] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3674247fb9b34a4bb13e38e6a33fde95, e87eb3fa5e0b44bca1d2a0ea3d0c1995; no voters:
I20260806 04:40:01.562674 8600 raft_consensus.cc:2781] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:40:01.562723 8600 raft_consensus.cc:482] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:40:01.562772 8600 raft_consensus.cc:3037] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:01.564098 8600 raft_consensus.cc:504] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.564502 8600 leader_election.cc:288] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [CANDIDATE]: Term 1 election: Requested vote from peers 3674247fb9b34a4bb13e38e6a33fde95 (127.31.166.130:35963), 755d861066ec45fdba645b4a4518d95b (127.31.166.131:41597)
I20260806 04:40:01.564844 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4ca5b8ca50431ba46071fac59d8880" candidate_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3674247fb9b34a4bb13e38e6a33fde95"
I20260806 04:40:01.564963 8362 raft_consensus.cc:3037] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:01.564946 8515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4ca5b8ca50431ba46071fac59d8880" candidate_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d861066ec45fdba645b4a4518d95b"
I20260806 04:40:01.565101 8515 raft_consensus.cc:3037] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:01.566045 8362 raft_consensus.cc:2454] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87eb3fa5e0b44bca1d2a0ea3d0c1995 in term 1.
I20260806 04:40:01.566133 8515 raft_consensus.cc:2454] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87eb3fa5e0b44bca1d2a0ea3d0c1995 in term 1.
I20260806 04:40:01.566440 8130 leader_election.cc:302] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3674247fb9b34a4bb13e38e6a33fde95, e87eb3fa5e0b44bca1d2a0ea3d0c1995; no voters:
I20260806 04:40:01.566712 8600 raft_consensus.cc:2781] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:40:01.567006 8600 raft_consensus.cc:686] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 LEADER]: Becoming Leader. State: Replica: e87eb3fa5e0b44bca1d2a0ea3d0c1995, State: Running, Role: LEADER
I20260806 04:40:01.567315 8600 consensus_queue.cc:237] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.569079 8646 tablet_replica.cc:439] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: TxnStatusTablet state changed. Reason: New leader e87eb3fa5e0b44bca1d2a0ea3d0c1995. Latest consensus state: current_term: 1 leader_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } }
I20260806 04:40:01.569188 8646 tablet_replica.cc:442] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:40:01.569514 8648 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 04:40:01.569700 8648 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 04:40:01.570300 8036 catalog_manager.cc:5495] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 reported cstate change: term changed from 0 to 1, leader changed from <none> to e87eb3fa5e0b44bca1d2a0ea3d0c1995 (127.31.166.129). New cstate: current_term: 1 leader_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } health_report { overall_health: UNKNOWN } } }
I20260806 04:40:01.588625 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.583366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:43819 (local address 127.0.0.1:37396)
0806 04:40:01.583584 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:01.583603 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:01.583707 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:01.584119 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:01.584122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:01.584128 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:40:01.584229 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.584233 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.585377 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.585380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.586000 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.586009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.586732 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.586753 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.586927 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.586993 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:40:01.588209 (+ 1216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:40:01.588231 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:40:01.588311 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 04:40:01.588424 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 04:40:01.588689 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.583543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.31.166.129:43819)
0806 04:40:01.583696 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.583701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.583720 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:40:01.583903 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.583906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.583980 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.584085 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:40:01.584338 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.585249 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.586119 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.586544 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.586680 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.587229 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:40:01.588078 (+ 849us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:40:01.588255 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:01.588375 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 04:40:01.588468 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:40:01.593794 8362 raft_consensus.cc:1261] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Refusing update from remote peer e87eb3fa5e0b44bca1d2a0ea3d0c1995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:40:01.593868 8515 raft_consensus.cc:1261] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Refusing update from remote peer e87eb3fa5e0b44bca1d2a0ea3d0c1995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:40:01.594439 8600 consensus_queue.cc:1035] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:40:01.594863 8600 consensus_queue.cc:1035] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:40:01.596495 8601 tablet_replica.cc:439] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95: TxnStatusTablet state changed. Reason: New leader e87eb3fa5e0b44bca1d2a0ea3d0c1995. Latest consensus state: current_term: 1 leader_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } }
I20260806 04:40:01.596621 8601 tablet_replica.cc:442] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:40:01.596798 8615 tablet_replica.cc:439] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b: TxnStatusTablet state changed. Reason: New leader e87eb3fa5e0b44bca1d2a0ea3d0c1995. Latest consensus state: current_term: 1 leader_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } }
I20260806 04:40:01.596910 8615 tablet_replica.cc:442] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:40:01.600909 8646 tablet_replica.cc:439] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } }
I20260806 04:40:01.601056 8646 tablet_replica.cc:442] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:40:01.604282 8615 tablet_replica.cc:439] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } }
I20260806 04:40:01.604417 8615 tablet_replica.cc:442] T 9c4ca5b8ca50431ba46071fac59d8880 P 755d861066ec45fdba645b4a4518d95b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:40:01.605425 8600 tablet_replica.cc:439] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } }
I20260806 04:40:01.605546 8600 tablet_replica.cc:442] T 9c4ca5b8ca50431ba46071fac59d8880 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: This TxnStatusTablet replica's current role is: LEADER
I20260806 04:40:01.606264 8601 tablet_replica.cc:439] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } } }
I20260806 04:40:01.606375 8601 tablet_replica.cc:442] T 9c4ca5b8ca50431ba46071fac59d8880 P 3674247fb9b34a4bb13e38e6a33fde95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 04:40:01.606565 8610 mvcc.cc:205] Tried to move back new op lower bound from 7315419961721888768 to 7315419961622450176. Current Snapshot: MvccSnapshot[applied={T|T < 7315419961721888768}]
I20260806 04:40:01.613511 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.609911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52862 (local address 127.31.166.130:35963)
0806 04:40:01.610030 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.610035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.610050 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:40:01.610119 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.610122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.610191 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.610304 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:40:01.610544 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.611197 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.612036 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.612275 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.612386 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.612680 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:40:01.613086 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:40:01.613207 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:01.613284 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 04:40:01.613342 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 04:40:01.626345 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.620847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:43363 (local address 127.31.166.129:43819)
0806 04:40:01.621021 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.621026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.623401 (+ 2375us) server_negotiation.cc:407] Connection header received
0806 04:40:01.623545 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.623550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.623601 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.623682 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:40:01.623925 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.624424 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.625273 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.625501 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.625610 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.625902 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:40:01.625971 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:40:01.626053 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:01.626142 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 04:40:01.626201 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:40:01.626570 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.620728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:43819 (local address 127.31.166.130:43363)
0806 04:40:01.623292 (+ 2564us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:01.623306 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:01.623429 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:01.623692 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:01.623695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:01.623703 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:40:01.623805 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.623813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.624536 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.624541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.625149 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.625157 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.625624 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.625628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.625718 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.625759 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:40:01.626059 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:40:01.626067 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 04:40:01.626132 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 04:40:01.626340 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2473,"thread_start_us":92,"threads_started":1}
I20260806 04:40:01.634187 8515 raft_consensus.cc:1261] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Refusing update from remote peer 3674247fb9b34a4bb13e38e6a33fde95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:40:01.634351 8205 raft_consensus.cc:1261] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Refusing update from remote peer 3674247fb9b34a4bb13e38e6a33fde95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:40:01.634725 8601 consensus_queue.cc:1035] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:40:01.635133 8630 consensus_queue.cc:1035] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:40:01.658221 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.653990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:58677 (local address 127.31.166.130:35963)
0806 04:40:01.654127 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.654132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.654415 (+ 283us) server_negotiation.cc:407] Connection header received
0806 04:40:01.654519 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.654522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.654572 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.654671 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:40:01.655030 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.655871 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.656898 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.657257 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.657358 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.657661 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:40:01.657744 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:40:01.657832 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:01.658027 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 04:40:01.658093 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:40:01.658298 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.653907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:35963 (local address 127.31.166.129:58677)
0806 04:40:01.654270 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:01.654286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:01.654395 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:01.654683 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:01.654686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:01.654694 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:40:01.654842 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.654850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.655970 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.655974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.656786 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:01.656796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.657392 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.657395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:01.657475 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.657520 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:40:01.657876 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:40:01.657882 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 04:40:01.658022 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 04:40:01.658136 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":145,"threads_started":1}
I20260806 04:40:01.686996 8033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:40:01.687222 8033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:40:01.689602 8033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615: Sending DeleteTablet for 3 replicas of tablet 6f914016c3114401a6c13ff48614b83c
I20260806 04:40:01.690222 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f914016c3114401a6c13ff48614b83c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:01 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37384
I20260806 04:40:01.690304 8495 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f914016c3114401a6c13ff48614b83c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:01 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:35852
I20260806 04:40:01.690335 8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f914016c3114401a6c13ff48614b83c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:01 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:52850
I20260806 04:40:01.690425 8620 tablet_replica.cc:330] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995: stopping tablet replica
I20260806 04:40:01.690531 8618 tablet_replica.cc:330] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95: stopping tablet replica
I20260806 04:40:01.690533 8620 raft_consensus.cc:2229] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:40:01.690668 8620 raft_consensus.cc:2258] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:40:01.690641 8618 raft_consensus.cc:2229] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:40:01.691025 8618 raft_consensus.cc:2258] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:40:01.691285 8622 tablet_replica.cc:330] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b: stopping tablet replica
I20260806 04:40:01.691427 8622 raft_consensus.cc:2229] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:40:01.691563 8622 raft_consensus.cc:2258] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:40:01.691895 8620 ts_tablet_manager.cc:1905] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:40:01.692509 8622 ts_tablet_manager.cc:1905] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:40:01.694885 8620 ts_tablet_manager.cc:1918] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:40:01.694957 8620 log.cc:1196] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/wal/wals/6f914016c3114401a6c13ff48614b83c
I20260806 04:40:01.695696 8622 ts_tablet_manager.cc:1918] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:40:01.695776 8622 log.cc:1196] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/6f914016c3114401a6c13ff48614b83c
I20260806 04:40:01.695946 8618 ts_tablet_manager.cc:1905] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:40:01.696048 8622 ts_tablet_manager.cc:1939] T 6f914016c3114401a6c13ff48614b83c P 755d861066ec45fdba645b4a4518d95b: Deleting consensus metadata
I20260806 04:40:01.696842 8020 catalog_manager.cc:4841] TS 755d861066ec45fdba645b4a4518d95b (127.31.166.131:41597): tablet 6f914016c3114401a6c13ff48614b83c (table test-table [id=b31d7d27ed6e4a1684b82266f3139eb9]) successfully deleted
I20260806 04:40:01.698890 8618 ts_tablet_manager.cc:1918] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:40:01.698966 8618 log.cc:1196] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/6f914016c3114401a6c13ff48614b83c
I20260806 04:40:01.699349 8620 ts_tablet_manager.cc:1939] T 6f914016c3114401a6c13ff48614b83c P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Deleting consensus metadata
I20260806 04:40:01.699616 8618 ts_tablet_manager.cc:1939] T 6f914016c3114401a6c13ff48614b83c P 3674247fb9b34a4bb13e38e6a33fde95: Deleting consensus metadata
I20260806 04:40:01.700496 8020 catalog_manager.cc:4841] TS e87eb3fa5e0b44bca1d2a0ea3d0c1995 (127.31.166.129:43819): tablet 6f914016c3114401a6c13ff48614b83c (table test-table [id=b31d7d27ed6e4a1684b82266f3139eb9]) successfully deleted
I20260806 04:40:01.701495 8018 catalog_manager.cc:4841] TS 3674247fb9b34a4bb13e38e6a33fde95 (127.31.166.130:35963): tablet 6f914016c3114401a6c13ff48614b83c (table test-table [id=b31d7d27ed6e4a1684b82266f3139eb9]) successfully deleted
Aug 06 04:40:01 dist-test-slave-vkjp krb5kdc[7984](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 04:40:01 dist-test-slave-vkjp krb5kdc[7984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:01 dist-test-slave-vkjp krb5kdc[7984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 04:40:01.729341 8638 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:01.721159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.31.166.190:37637)
0806 04:40:01.721302 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.721307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.721529 (+ 222us) server_negotiation.cc:407] Connection header received
0806 04:40:01.721666 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.721669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.721728 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.721834 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:01.722726 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.723606 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.724851 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.725219 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.725322 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.729198 (+ 3876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36654: BlockingRecv error: recv got EOF from 127.0.0.1:36654 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 06 04:40:01 dist-test-slave-vkjp krb5kdc[7984](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM
I20260806 04:40:01.742065 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.731281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.31.166.190:37637)
0806 04:40:01.731388 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.731393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.731663 (+ 270us) server_negotiation.cc:407] Connection header received
0806 04:40:01.731850 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.731854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.731904 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.732007 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:01.732753 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.733547 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.734769 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.735181 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.735290 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.738313 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:01.738318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:01.738321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:01.738349 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:01.740128 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:01.740647 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:01.740650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:01.740652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:01.740745 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:01.741089 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:01.741092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:01.741094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:01.741320 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:01.741456 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:01.741748 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 04:40:01.741886 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:40:01.745154 8033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:40:01.745635 8033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:40:01.750728 8183 tablet_service.cc:1467] Processing CreateTablet for tablet a23ea05b33314e60944c6d921c769417 (DEFAULT_TABLE table=test-table [id=8ecbdf1cd6324608a2a5c9c830ea6d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:01.750845 8341 tablet_service.cc:1467] Processing CreateTablet for tablet a23ea05b33314e60944c6d921c769417 (DEFAULT_TABLE table=test-table [id=8ecbdf1cd6324608a2a5c9c830ea6d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:01.751158 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a23ea05b33314e60944c6d921c769417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:01.751374 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a23ea05b33314e60944c6d921c769417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:01.753798 8596 tablet_bootstrap.cc:492] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Bootstrap starting.
I20260806 04:40:01.753850 8594 tablet_bootstrap.cc:492] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95: Bootstrap starting.
I20260806 04:40:01.754855 8596 tablet_bootstrap.cc:654] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:01.754855 8594 tablet_bootstrap.cc:654] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:01.756738 8596 tablet_bootstrap.cc:492] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: No bootstrap required, opened a new log
I20260806 04:40:01.756839 8596 ts_tablet_manager.cc:1397] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 04:40:01.757150 8495 tablet_service.cc:1467] Processing CreateTablet for tablet a23ea05b33314e60944c6d921c769417 (DEFAULT_TABLE table=test-table [id=8ecbdf1cd6324608a2a5c9c830ea6d12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:01.757289 8596 raft_consensus.cc:348] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.757418 8596 raft_consensus.cc:374] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:01.757455 8596 raft_consensus.cc:729] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87eb3fa5e0b44bca1d2a0ea3d0c1995, State: Initialized, Role: FOLLOWER
I20260806 04:40:01.757479 8495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a23ea05b33314e60944c6d921c769417. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:01.757589 8596 consensus_queue.cc:260] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.757889 8596 ts_tablet_manager.cc:1428] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260806 04:40:01.760289 8595 tablet_bootstrap.cc:492] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b: Bootstrap starting.
I20260806 04:40:01.761292 8595 tablet_bootstrap.cc:654] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:01.761406 8594 tablet_bootstrap.cc:492] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95: No bootstrap required, opened a new log
I20260806 04:40:01.761498 8594 ts_tablet_manager.cc:1397] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260806 04:40:01.762022 8594 raft_consensus.cc:348] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.762164 8594 raft_consensus.cc:374] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:01.762204 8594 raft_consensus.cc:729] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3674247fb9b34a4bb13e38e6a33fde95, State: Initialized, Role: FOLLOWER
I20260806 04:40:01.763015 8595 tablet_bootstrap.cc:492] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b: No bootstrap required, opened a new log
I20260806 04:40:01.763123 8595 ts_tablet_manager.cc:1397] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:40:01.763288 8594 consensus_queue.cc:260] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.763605 8594 ts_tablet_manager.cc:1428] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 04:40:01.763540 8595 raft_consensus.cc:348] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.763648 8595 raft_consensus.cc:374] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:01.763680 8595 raft_consensus.cc:729] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d861066ec45fdba645b4a4518d95b, State: Initialized, Role: FOLLOWER
I20260806 04:40:01.763808 8595 consensus_queue.cc:260] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.764071 8595 ts_tablet_manager.cc:1428] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:40:01.772868 8630 raft_consensus.cc:482] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:40:01.773031 8630 raft_consensus.cc:504] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.773509 8630 leader_election.cc:288] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e87eb3fa5e0b44bca1d2a0ea3d0c1995 (127.31.166.129:43819), 755d861066ec45fdba645b4a4518d95b (127.31.166.131:41597)
I20260806 04:40:01.773978 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23ea05b33314e60944c6d921c769417" candidate_uuid: "3674247fb9b34a4bb13e38e6a33fde95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" is_pre_election: true
I20260806 04:40:01.774159 8205 raft_consensus.cc:2454] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3674247fb9b34a4bb13e38e6a33fde95 in term 0.
I20260806 04:40:01.774443 8515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23ea05b33314e60944c6d921c769417" candidate_uuid: "3674247fb9b34a4bb13e38e6a33fde95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d861066ec45fdba645b4a4518d95b" is_pre_election: true
I20260806 04:40:01.774518 8284 leader_election.cc:302] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3674247fb9b34a4bb13e38e6a33fde95, e87eb3fa5e0b44bca1d2a0ea3d0c1995; no voters:
I20260806 04:40:01.774598 8515 raft_consensus.cc:2454] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3674247fb9b34a4bb13e38e6a33fde95 in term 0.
I20260806 04:40:01.774911 8630 raft_consensus.cc:2781] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:40:01.774979 8630 raft_consensus.cc:482] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:40:01.775038 8630 raft_consensus.cc:3037] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:01.776378 8630 raft_consensus.cc:504] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.776739 8630 leader_election.cc:288] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [CANDIDATE]: Term 1 election: Requested vote from peers e87eb3fa5e0b44bca1d2a0ea3d0c1995 (127.31.166.129:43819), 755d861066ec45fdba645b4a4518d95b (127.31.166.131:41597)
I20260806 04:40:01.777228 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23ea05b33314e60944c6d921c769417" candidate_uuid: "3674247fb9b34a4bb13e38e6a33fde95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995"
I20260806 04:40:01.777365 8205 raft_consensus.cc:3037] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:01.777340 8515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a23ea05b33314e60944c6d921c769417" candidate_uuid: "3674247fb9b34a4bb13e38e6a33fde95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d861066ec45fdba645b4a4518d95b"
I20260806 04:40:01.777462 8515 raft_consensus.cc:3037] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:01.778733 8205 raft_consensus.cc:2454] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3674247fb9b34a4bb13e38e6a33fde95 in term 1.
I20260806 04:40:01.778731 8515 raft_consensus.cc:2454] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3674247fb9b34a4bb13e38e6a33fde95 in term 1.
I20260806 04:40:01.779122 8284 leader_election.cc:302] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3674247fb9b34a4bb13e38e6a33fde95, e87eb3fa5e0b44bca1d2a0ea3d0c1995; no voters:
I20260806 04:40:01.779333 8630 raft_consensus.cc:2781] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:40:01.779443 8630 raft_consensus.cc:686] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 LEADER]: Becoming Leader. State: Replica: 3674247fb9b34a4bb13e38e6a33fde95, State: Running, Role: LEADER
I20260806 04:40:01.779592 8630 consensus_queue.cc:237] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.781519 8033 catalog_manager.cc:5495] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3674247fb9b34a4bb13e38e6a33fde95 (127.31.166.130). New cstate: current_term: 1 leader_uuid: "3674247fb9b34a4bb13e38e6a33fde95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } health_report { overall_health: UNKNOWN } } }
I20260806 04:40:01.795691 8592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:01.791923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52868 (local address 127.31.166.130:35963)
0806 04:40:01.792062 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:01.792066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:01.792080 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:40:01.792226 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:01.792229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:01.792275 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:01.792351 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:40:01.792664 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.793172 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.794437 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:01.794659 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:01.794750 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:01.795141 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:40:01.795242 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:40:01.795326 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:01.795466 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 04:40:01.795518 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:40:01.797923 8205 raft_consensus.cc:1261] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Refusing update from remote peer 3674247fb9b34a4bb13e38e6a33fde95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:40:01.797988 8515 raft_consensus.cc:1261] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Refusing update from remote peer 3674247fb9b34a4bb13e38e6a33fde95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:40:01.798440 8663 consensus_queue.cc:1035] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:40:01.798624 8630 consensus_queue.cc:1035] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:40:01.809718 8034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:40:01.809906 8034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:40:01.811985 8034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615: Sending DeleteTablet for 3 replicas of tablet a23ea05b33314e60944c6d921c769417
I20260806 04:40:01.812610 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet a23ea05b33314e60944c6d921c769417 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:01 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37384
I20260806 04:40:01.812621 8341 tablet_service.cc:1514] Processing DeleteTablet for tablet a23ea05b33314e60944c6d921c769417 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:01 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:52850
I20260806 04:40:01.812822 8620 tablet_replica.cc:330] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: stopping tablet replica
I20260806 04:40:01.812825 8618 tablet_replica.cc:330] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95: stopping tablet replica
I20260806 04:40:01.812893 8495 tablet_service.cc:1514] Processing DeleteTablet for tablet a23ea05b33314e60944c6d921c769417 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:01 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:35852
I20260806 04:40:01.812927 8620 raft_consensus.cc:2229] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:40:01.812948 8618 raft_consensus.cc:2229] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:40:01.813055 8620 raft_consensus.cc:2258] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:40:01.813066 8622 tablet_replica.cc:330] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b: stopping tablet replica
I20260806 04:40:01.813169 8622 raft_consensus.cc:2229] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:40:01.813176 8618 raft_consensus.cc:2258] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:40:01.813298 8622 raft_consensus.cc:2258] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:40:01.814297 8618 ts_tablet_manager.cc:1905] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:40:01.814395 8622 ts_tablet_manager.cc:1905] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:40:01.814531 8034 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:40:01.815172 8034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:40:01.817111 8622 ts_tablet_manager.cc:1918] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:40:01.817201 8622 log.cc:1196] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/a23ea05b33314e60944c6d921c769417
I20260806 04:40:01.817181 8618 ts_tablet_manager.cc:1918] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:40:01.817257 8618 log.cc:1196] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/a23ea05b33314e60944c6d921c769417
I20260806 04:40:01.817490 8622 ts_tablet_manager.cc:1939] T a23ea05b33314e60944c6d921c769417 P 755d861066ec45fdba645b4a4518d95b: Deleting consensus metadata
I20260806 04:40:01.817564 8618 ts_tablet_manager.cc:1939] T a23ea05b33314e60944c6d921c769417 P 3674247fb9b34a4bb13e38e6a33fde95: Deleting consensus metadata
I20260806 04:40:01.818790 8018 catalog_manager.cc:4841] TS 3674247fb9b34a4bb13e38e6a33fde95 (127.31.166.130:35963): tablet a23ea05b33314e60944c6d921c769417 (table test-table [id=8ecbdf1cd6324608a2a5c9c830ea6d12]) successfully deleted
I20260806 04:40:01.818826 8020 catalog_manager.cc:4841] TS 755d861066ec45fdba645b4a4518d95b (127.31.166.131:41597): tablet a23ea05b33314e60944c6d921c769417 (table test-table [id=8ecbdf1cd6324608a2a5c9c830ea6d12]) successfully deleted
I20260806 04:40:01.819674 8620 ts_tablet_manager.cc:1905] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:40:01.820623 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 613926504fa840588e7133a2f070eaa1 (DEFAULT_TABLE table=test-table [id=fdf0ae38ca1b415cbeabb9661b2a53c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:01.820942 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613926504fa840588e7133a2f070eaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:01.821039 8495 tablet_service.cc:1467] Processing CreateTablet for tablet 613926504fa840588e7133a2f070eaa1 (DEFAULT_TABLE table=test-table [id=fdf0ae38ca1b415cbeabb9661b2a53c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:01.821288 8495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613926504fa840588e7133a2f070eaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:01.822396 8620 ts_tablet_manager.cc:1918] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:40:01.822474 8620 log.cc:1196] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-0/wal/wals/a23ea05b33314e60944c6d921c769417
I20260806 04:40:01.822758 8620 ts_tablet_manager.cc:1939] T a23ea05b33314e60944c6d921c769417 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Deleting consensus metadata
I20260806 04:40:01.823503 8596 tablet_bootstrap.cc:492] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Bootstrap starting.
I20260806 04:40:01.823760 8595 tablet_bootstrap.cc:492] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b: Bootstrap starting.
I20260806 04:40:01.824402 8596 tablet_bootstrap.cc:654] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:01.824532 8595 tablet_bootstrap.cc:654] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:01.825153 8020 catalog_manager.cc:4841] TS e87eb3fa5e0b44bca1d2a0ea3d0c1995 (127.31.166.129:43819): tablet a23ea05b33314e60944c6d921c769417 (table test-table [id=8ecbdf1cd6324608a2a5c9c830ea6d12]) successfully deleted
I20260806 04:40:01.825536 8341 tablet_service.cc:1467] Processing CreateTablet for tablet 613926504fa840588e7133a2f070eaa1 (DEFAULT_TABLE table=test-table [id=fdf0ae38ca1b415cbeabb9661b2a53c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:01.825842 8341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613926504fa840588e7133a2f070eaa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:01.826363 8595 tablet_bootstrap.cc:492] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b: No bootstrap required, opened a new log
I20260806 04:40:01.826436 8595 ts_tablet_manager.cc:1397] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260806 04:40:01.826817 8595 raft_consensus.cc:348] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.826938 8595 raft_consensus.cc:374] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:01.826968 8595 raft_consensus.cc:729] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 755d861066ec45fdba645b4a4518d95b, State: Initialized, Role: FOLLOWER
I20260806 04:40:01.827136 8595 consensus_queue.cc:260] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.827438 8595 ts_tablet_manager.cc:1428] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:40:01.828900 8594 tablet_bootstrap.cc:492] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95: Bootstrap starting.
I20260806 04:40:01.829973 8594 tablet_bootstrap.cc:654] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:01.831528 8594 tablet_bootstrap.cc:492] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95: No bootstrap required, opened a new log
I20260806 04:40:01.831629 8594 ts_tablet_manager.cc:1397] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:40:01.832108 8594 raft_consensus.cc:348] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.832230 8594 raft_consensus.cc:374] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:01.832275 8594 raft_consensus.cc:729] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3674247fb9b34a4bb13e38e6a33fde95, State: Initialized, Role: FOLLOWER
I20260806 04:40:01.832397 8594 consensus_queue.cc:260] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.832662 8594 ts_tablet_manager.cc:1428] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:40:01.834722 8596 tablet_bootstrap.cc:492] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: No bootstrap required, opened a new log
I20260806 04:40:01.834837 8596 ts_tablet_manager.cc:1397] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20260806 04:40:01.835367 8596 raft_consensus.cc:348] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.835496 8596 raft_consensus.cc:374] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:01.835542 8596 raft_consensus.cc:729] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e87eb3fa5e0b44bca1d2a0ea3d0c1995, State: Initialized, Role: FOLLOWER
I20260806 04:40:01.835671 8596 consensus_queue.cc:260] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:01.835979 8596 ts_tablet_manager.cc:1428] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:40:02.052886 8600 raft_consensus.cc:482] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:40:02.053066 8600 raft_consensus.cc:504] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:02.053565 8600 leader_election.cc:288] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3674247fb9b34a4bb13e38e6a33fde95 (127.31.166.130:35963), 755d861066ec45fdba645b4a4518d95b (127.31.166.131:41597)
I20260806 04:40:02.054164 8515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613926504fa840588e7133a2f070eaa1" candidate_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d861066ec45fdba645b4a4518d95b" is_pre_election: true
I20260806 04:40:02.054183 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613926504fa840588e7133a2f070eaa1" candidate_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3674247fb9b34a4bb13e38e6a33fde95" is_pre_election: true
I20260806 04:40:02.054327 8515 raft_consensus.cc:2454] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87eb3fa5e0b44bca1d2a0ea3d0c1995 in term 0.
I20260806 04:40:02.054437 8362 raft_consensus.cc:2454] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e87eb3fa5e0b44bca1d2a0ea3d0c1995 in term 0.
I20260806 04:40:02.054673 8132 leader_election.cc:302] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d861066ec45fdba645b4a4518d95b, e87eb3fa5e0b44bca1d2a0ea3d0c1995; no voters:
I20260806 04:40:02.054884 8600 raft_consensus.cc:2781] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:40:02.054942 8600 raft_consensus.cc:482] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:40:02.054970 8600 raft_consensus.cc:3037] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:02.055954 8600 raft_consensus.cc:504] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:02.056283 8600 leader_election.cc:288] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [CANDIDATE]: Term 1 election: Requested vote from peers 3674247fb9b34a4bb13e38e6a33fde95 (127.31.166.130:35963), 755d861066ec45fdba645b4a4518d95b (127.31.166.131:41597)
I20260806 04:40:02.056640 8362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613926504fa840588e7133a2f070eaa1" candidate_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3674247fb9b34a4bb13e38e6a33fde95"
I20260806 04:40:02.056702 8515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613926504fa840588e7133a2f070eaa1" candidate_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "755d861066ec45fdba645b4a4518d95b"
I20260806 04:40:02.056749 8362 raft_consensus.cc:3037] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:02.056847 8515 raft_consensus.cc:3037] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:02.057868 8362 raft_consensus.cc:2454] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87eb3fa5e0b44bca1d2a0ea3d0c1995 in term 1.
I20260806 04:40:02.057953 8515 raft_consensus.cc:2454] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e87eb3fa5e0b44bca1d2a0ea3d0c1995 in term 1.
I20260806 04:40:02.058267 8132 leader_election.cc:302] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 755d861066ec45fdba645b4a4518d95b, e87eb3fa5e0b44bca1d2a0ea3d0c1995; no voters:
I20260806 04:40:02.058473 8600 raft_consensus.cc:2781] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:40:02.058575 8600 raft_consensus.cc:686] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 LEADER]: Becoming Leader. State: Replica: e87eb3fa5e0b44bca1d2a0ea3d0c1995, State: Running, Role: LEADER
I20260806 04:40:02.058729 8600 consensus_queue.cc:237] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } }
I20260806 04:40:02.060667 8034 catalog_manager.cc:5495] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 reported cstate change: term changed from 0 to 1, leader changed from <none> to e87eb3fa5e0b44bca1d2a0ea3d0c1995 (127.31.166.129). New cstate: current_term: 1 leader_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e87eb3fa5e0b44bca1d2a0ea3d0c1995" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 43819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 } health_report { overall_health: UNKNOWN } } }
I20260806 04:40:02.082094 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:02.078919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37402 (local address 127.31.166.129:43819)
0806 04:40:02.079034 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:02.079037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:02.079050 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:40:02.079217 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:02.079220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:02.079272 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:02.079343 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:40:02.079604 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:02.080237 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:02.081048 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:02.081349 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:02.081434 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:02.081663 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:40:02.081730 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:40:02.081820 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:02.081913 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 04:40:02.081961 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 04:40:02.091701 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:02.088375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:34213 (local address 127.31.166.129:43819)
0806 04:40:02.088510 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:02.088515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:02.088529 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:40:02.088673 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:02.088676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:02.088722 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:02.088823 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:40:02.089091 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:02.089670 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:02.090558 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:02.090793 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:02.090909 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:02.091240 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:40:02.091320 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:40:02.091376 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:02.091520 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 04:40:02.091576 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":61}
I20260806 04:40:02.091735 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:02.088139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:43819 (local address 127.31.166.129:34213)
0806 04:40:02.088423 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:02.088443 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:02.088546 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:02.088830 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:02.088834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:02.088840 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 04:40:02.088965 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:02.088971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:02.089793 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:02.089800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:02.090429 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:02.090437 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:02.090909 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:02.090913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:02.091012 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:02.091045 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 04:40:02.091410 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 04:40:02.091418 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 04:40:02.091495 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 04:40:02.091589 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":7}
I20260806 04:40:02.098176 8362 raft_consensus.cc:1261] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Refusing update from remote peer e87eb3fa5e0b44bca1d2a0ea3d0c1995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:40:02.098520 8515 raft_consensus.cc:1261] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Refusing update from remote peer e87eb3fa5e0b44bca1d2a0ea3d0c1995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:40:02.098613 8657 consensus_queue.cc:1035] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3674247fb9b34a4bb13e38e6a33fde95" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 35963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:40:02.099009 8657 consensus_queue.cc:1035] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "755d861066ec45fdba645b4a4518d95b" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:40:02.136840 8033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:40:02.137059 8033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:40:02.139446 8033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d60c3ffdadb4b4cb2bddffb12723615: Sending DeleteTablet for 3 replicas of tablet 613926504fa840588e7133a2f070eaa1
I20260806 04:40:02.140166 8495 tablet_service.cc:1514] Processing DeleteTablet for tablet 613926504fa840588e7133a2f070eaa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:02 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:35852
I20260806 04:40:02.140232 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet 613926504fa840588e7133a2f070eaa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:02 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37384
I20260806 04:40:02.140388 8622 tablet_replica.cc:330] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b: stopping tablet replica
I20260806 04:40:02.140439 8620 tablet_replica.cc:330] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: stopping tablet replica
I20260806 04:40:02.140530 8622 raft_consensus.cc:2229] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:40:02.140551 8620 raft_consensus.cc:2229] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:40:02.140666 8622 raft_consensus.cc:2258] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:40:02.140878 8620 raft_consensus.cc:2258] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:40:02.141619 8622 ts_tablet_manager.cc:1905] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:40:02.141983 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 8113
I20260806 04:40:02.142270 8620 ts_tablet_manager.cc:1905] T 613926504fa840588e7133a2f070eaa1 P e87eb3fa5e0b44bca1d2a0ea3d0c1995: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:40:02.142918 8341 tablet_service.cc:1514] Processing DeleteTablet for tablet 613926504fa840588e7133a2f070eaa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:02 UTC) from {username='oryx', principal='oryx/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:52850
I20260806 04:40:02.143162 8618 tablet_replica.cc:330] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95: stopping tablet replica
I20260806 04:40:02.143342 8618 raft_consensus.cc:2229] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:40:02.143486 8618 raft_consensus.cc:2258] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:40:02.144397 8618 ts_tablet_manager.cc:1905] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:40:02.144989 8622 ts_tablet_manager.cc:1918] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:40:02.145074 8622 log.cc:1196] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/613926504fa840588e7133a2f070eaa1
I20260806 04:40:02.145315 8622 ts_tablet_manager.cc:1939] T 613926504fa840588e7133a2f070eaa1 P 755d861066ec45fdba645b4a4518d95b: Deleting consensus metadata
I20260806 04:40:02.146482 8020 catalog_manager.cc:4841] TS 755d861066ec45fdba645b4a4518d95b (127.31.166.131:41597): tablet 613926504fa840588e7133a2f070eaa1 (table test-table [id=fdf0ae38ca1b415cbeabb9661b2a53c9]) successfully deleted
I20260806 04:40:02.146997 8618 ts_tablet_manager.cc:1918] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 04:40:02.147096 8618 log.cc:1196] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/613926504fa840588e7133a2f070eaa1
I20260806 04:40:02.147403 8618 ts_tablet_manager.cc:1939] T 613926504fa840588e7133a2f070eaa1 P 3674247fb9b34a4bb13e38e6a33fde95: Deleting consensus metadata
I20260806 04:40:02.148102 8018 catalog_manager.cc:4841] TS 3674247fb9b34a4bb13e38e6a33fde95 (127.31.166.130:35963): tablet 613926504fa840588e7133a2f070eaa1 (table test-table [id=fdf0ae38ca1b415cbeabb9661b2a53c9]) successfully deleted
W20260806 04:40:02.153267 8020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:43819 (error 108)
I20260806 04:40:02.153378 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 8263
I20260806 04:40:02.156208 8695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:02.155661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:43819 (local address 127.0.0.1:37408)
0806 04:40:02.155973 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:02.156105 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:43819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
W20260806 04:40:02.156373 8020 catalog_manager.cc:4584] TS e87eb3fa5e0b44bca1d2a0ea3d0c1995 (127.31.166.129:43819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 613926504fa840588e7133a2f070eaa1: Network error: Client connection negotiation failed: client connection to 127.31.166.129:43819: connect: Connection refused (error 111)
I20260806 04:40:02.162716 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 8422
I20260806 04:40:02.171257 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 7999
2026-08-06T04:40:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3567 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:40:02 dist-test-slave-vkjp krb5kdc[8698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:40:02 dist-test-slave-vkjp krb5kdc[8698](info): set up 2 sockets
Aug 06 04:40:02 dist-test-slave-vkjp krb5kdc[8698](info): commencing operation
krb5kdc: starting...
W20260806 04:40:03.305763 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:40:03 dist-test-slave-vkjp krb5kdc[8698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:40:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:40:03.430462 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:34007
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:34007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:40:03.539611 8713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:03.539880 8713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:03.539921 8713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:03.543833 8713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:40:03.543888 8713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:03.543907 8713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:03.543923 8713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:40:03.543938 8713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:40:03.549151 8713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:34007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:34007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:03.550146 8713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:03.551111 8713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:03.559350 8720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:03.559379 8721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:03.559350 8723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:03.559679 8713 server_base.cc:1034] running on GCE node
I20260806 04:40:03.560319 8713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:03.561465 8713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:03.562667 8713 hybrid_clock.cc:648] HybridClock initialized: now 1785991203562648 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T04:40:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1404 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:40:03 dist-test-slave-vkjp krb5kdc[8728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:40:03 dist-test-slave-vkjp krb5kdc[8728](info): set up 2 sockets
Aug 06 04:40:03 dist-test-slave-vkjp krb5kdc[8728](info): commencing operation
krb5kdc: starting...
W20260806 04:40:04.699721 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:40:04 dist-test-slave-vkjp krb5kdc[8728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:40:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:40:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:40:04.830049 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:40931
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:40931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:40:04.943373 8742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:04.943673 8742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:04.943768 8742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:04.948174 8742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:40:04.948252 8742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:04.948279 8742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:04.948298 8742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:40:04.948374 8742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:40:04.953917 8742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:40931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:40931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:04.955135 8742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:04.956316 8742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:04.964977 8750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:04.964977 8749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:04.964993 8753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:04.965301 8742 server_base.cc:1034] running on GCE node
I20260806 04:40:04.965984 8742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:04.967073 8742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:04.968286 8742 hybrid_clock.cc:648] HybridClock initialized: now 1785991204968276 us; error 30 us; skew 500 ppm
Aug 06 04:40:04 dist-test-slave-vkjp krb5kdc[8728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991204, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:04.971230 8742 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:40:04.972191 8742 webserver.cc:460] Webserver started at http://127.31.166.190:43913/ using document root <none> and password file <none>
I20260806 04:40:04.972838 8742 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:04.972956 8742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:04.973197 8742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:04.975502 8742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "82f0657b71ce4fb9ba17811490530492"
format_stamp: "Formatted at 2026-08-06 04:40:04 on dist-test-slave-vkjp"
server_key: "445b5c9d36e295d055128c889b67ce30"
server_key_iv: "82cdcb8440dad10902e674f3dc4ac3d1"
server_key_version: "encryptionkey@0"
I20260806 04:40:04.976104 8742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "82f0657b71ce4fb9ba17811490530492"
format_stamp: "Formatted at 2026-08-06 04:40:04 on dist-test-slave-vkjp"
server_key: "445b5c9d36e295d055128c889b67ce30"
server_key_iv: "82cdcb8440dad10902e674f3dc4ac3d1"
server_key_version: "encryptionkey@0"
I20260806 04:40:04.980271 8742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:40:04.982772 8760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:04.984072 8742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:40:04.984218 8742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "82f0657b71ce4fb9ba17811490530492"
format_stamp: "Formatted at 2026-08-06 04:40:04 on dist-test-slave-vkjp"
server_key: "445b5c9d36e295d055128c889b67ce30"
server_key_iv: "82cdcb8440dad10902e674f3dc4ac3d1"
server_key_version: "encryptionkey@0"
I20260806 04:40:04.984329 8742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:05.003801 8742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:05.007287 8742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:05.007498 8742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:05.014825 8742 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:40931
I20260806 04:40:05.014863 8817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:40931 every 8 connection(s)
I20260806 04:40:05.015829 8742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:40:05.016739 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 8742
I20260806 04:40:05.016908 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:40:05.017238 32410 external_mini_cluster.cc:1428] Setting key 6e7176b71cc8bffa7f38a6a2b14de41a
I20260806 04:40:05.019233 8818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:05.024855 8818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492: Bootstrap starting.
Aug 06 04:40:05 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:40:05.027393 8818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:05.028146 8818 log.cc:824] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492: Log is configured to *not* fsync() on all Append() calls
I20260806 04:40:05.030453 8818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492: No bootstrap required, opened a new log
I20260806 04:40:05.032974 8822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:05.018801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51864 (local address 127.31.166.190:40931)
0806 04:40:05.019324 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:05.019346 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:05.019399 (+ 53us) server_negotiation.cc:407] Connection header received
0806 04:40:05.020117 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:05.020133 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:05.020480 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:05.020866 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:05.021744 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:05.022862 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:05.023883 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:05.024394 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:05.024580 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:05.027578 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:05.027583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:05.027596 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:05.027633 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:05.029626 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:05.030180 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:05.030185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:05.030189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:05.030307 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:05.030642 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:05.030646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:05.030647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:05.031189 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:05.031381 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:05.031643 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 04:40:05.031944 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":146,"threads_started":1}
I20260806 04:40:05.034890 8818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f0657b71ce4fb9ba17811490530492" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40931 } }
I20260806 04:40:05.035202 8818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:05.035306 8818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f0657b71ce4fb9ba17811490530492, State: Initialized, Role: FOLLOWER
I20260806 04:40:05.035712 8818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f0657b71ce4fb9ba17811490530492" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40931 } }
I20260806 04:40:05.035840 8818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:40:05.035883 8818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:40:05.035957 8818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:05.036926 8818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f0657b71ce4fb9ba17811490530492" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40931 } }
I20260806 04:40:05.037243 8818 leader_election.cc:302] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82f0657b71ce4fb9ba17811490530492; no voters:
I20260806 04:40:05.037554 8818 leader_election.cc:288] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:40:05.037854 8824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:40:05.038084 8824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 1 LEADER]: Becoming Leader. State: Replica: 82f0657b71ce4fb9ba17811490530492, State: Running, Role: LEADER
I20260806 04:40:05.038405 8824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f0657b71ce4fb9ba17811490530492" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40931 } }
I20260806 04:40:05.038727 8818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:40:05.040247 8825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82f0657b71ce4fb9ba17811490530492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f0657b71ce4fb9ba17811490530492" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40931 } } }
I20260806 04:40:05.040331 8826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82f0657b71ce4fb9ba17811490530492. Latest consensus state: current_term: 1 leader_uuid: "82f0657b71ce4fb9ba17811490530492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f0657b71ce4fb9ba17811490530492" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40931 } } }
I20260806 04:40:05.040409 8825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [sys.catalog]: This master's current role is: LEADER
I20260806 04:40:05.040409 8826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [sys.catalog]: This master's current role is: LEADER
W20260806 04:40:05.043669 8841 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:40:05.043764 8841 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:40:05.043896 8838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:40:05.045022 8838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:40:05.050406 8838 catalog_manager.cc:1347] Generated new cluster ID: d704157390c342fab359fee7ca53ccba
I20260806 04:40:05.050494 8838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:40:05.083514 8838 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:40:05.084563 8838 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:40:05.099121 8838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492: Generated new TSK 0
I20260806 04:40:05.099998 8838 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:40:05.138631 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:40931
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:40:05.248993 8847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:05.249251 8847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:05.249306 8847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:05.253278 8847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:05.253335 8847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:05.253424 8847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:40:05.258567 8847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:05.259652 8847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:05.260694 8847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:05.270468 8853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:05.270396 8852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:05.270391 8855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:05.270750 8847 server_base.cc:1034] running on GCE node
I20260806 04:40:05.271236 8847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:05.271862 8847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:05.273049 8847 hybrid_clock.cc:648] HybridClock initialized: now 1785991205273035 us; error 26 us; skew 500 ppm
Aug 06 04:40:05 dist-test-slave-vkjp krb5kdc[8728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991205, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:05.275799 8847 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:40:05.276688 8847 webserver.cc:460] Webserver started at http://127.31.166.129:34521/ using document root <none> and password file <none>
I20260806 04:40:05.277251 8847 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:05.277340 8847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:05.277582 8847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:05.279665 8847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "c2d0e9edf9da4a708307eba3022dc64e"
format_stamp: "Formatted at 2026-08-06 04:40:05 on dist-test-slave-vkjp"
server_key: "74d5aaa37fc527cec951e4124550093a"
server_key_iv: "adb6048091b272691ef8ad94b84222b1"
server_key_version: "encryptionkey@0"
I20260806 04:40:05.280226 8847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "c2d0e9edf9da4a708307eba3022dc64e"
format_stamp: "Formatted at 2026-08-06 04:40:05 on dist-test-slave-vkjp"
server_key: "74d5aaa37fc527cec951e4124550093a"
server_key_iv: "adb6048091b272691ef8ad94b84222b1"
server_key_version: "encryptionkey@0"
I20260806 04:40:05.283825 8847 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 04:40:05.286284 8864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:05.287647 8847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:40:05.287775 8847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "c2d0e9edf9da4a708307eba3022dc64e"
format_stamp: "Formatted at 2026-08-06 04:40:05 on dist-test-slave-vkjp"
server_key: "74d5aaa37fc527cec951e4124550093a"
server_key_iv: "adb6048091b272691ef8ad94b84222b1"
server_key_version: "encryptionkey@0"
I20260806 04:40:05.287892 8847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:05.309077 8847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:05.312280 8847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:05.312513 8847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:05.313133 8847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:05.314003 8847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:05.314071 8847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:05.314141 8847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:05.314186 8847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:05.323858 8847 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:37233
I20260806 04:40:05.323882 8988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:37233 every 8 connection(s)
I20260806 04:40:05.324890 8847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:40:05.325912 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 8847
I20260806 04:40:05.326021 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:40:05.326278 32410 external_mini_cluster.cc:1428] Setting key 5eff808955ef0de4e37bce386f7a2310
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
Aug 06 04:40:05 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991205, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
I20260806 04:40:05.340862 8822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:05.327946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:42253 (local address 127.31.166.190:40931)
0806 04:40:05.328142 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:05.328148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:05.329011 (+ 863us) server_negotiation.cc:407] Connection header received
0806 04:40:05.330385 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:05.330388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:05.330442 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:05.330545 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:05.331960 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:05.332603 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:05.333742 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:05.333976 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:05.334089 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:05.336630 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:05.336633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:05.336636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:05.336667 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:05.338064 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:05.338677 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:05.338684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:05.338688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:05.338809 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:05.339303 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:05.339308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:05.339310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:05.339578 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:05.339694 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:05.340475 (+ 781us) server_negotiation.cc:299] Negotiation successful
0806 04:40:05.340644 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 04:40:05.342080 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:05.328334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:42253)
0806 04:40:05.328832 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:05.328898 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:05.330171 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:05.330726 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:05.330733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:05.331177 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:05.331775 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:05.331792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:05.332730 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:05.332733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:05.333614 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:05.333623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:05.334133 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:05.334137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:05.334316 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:05.334931 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:05.334949 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:05.336456 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:05.338251 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:05.338257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:05.338270 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:05.338549 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:05.338947 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:05.338954 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:05.338957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:05.339181 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:05.339817 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:05.339827 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:05.340192 (+ 365us) client_negotiation.cc:769] Sending connection context
0806 04:40:05.340381 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 04:40:05.340822 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":146,"threads_started":1}
I20260806 04:40:05.343443 8989 heartbeater.cc:347] Connected to a master server at 127.31.166.190:40931
I20260806 04:40:05.343691 8989 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:05.344188 8989 heartbeater.cc:511] Master 127.31.166.190:40931 requested a full tablet report, sending...
I20260806 04:40:05.345794 8779 ts_manager.cc:194] Registered new tserver with Master: c2d0e9edf9da4a708307eba3022dc64e (127.31.166.129:37233)
I20260806 04:40:05.346900 8779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:42253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:40:05.358094 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:40931
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:40:05.473608 8995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:05.473964 8995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:05.474052 8995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:05.478175 8995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:05.478269 8995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:05.478371 8995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:40:05.483754 8995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:05.484853 8995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:05.485821 8995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:05.495554 9005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:05.495544 9002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:05.495515 9003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:05.495846 8995 server_base.cc:1034] running on GCE node
I20260806 04:40:05.496238 8995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:05.496944 8995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:05.498122 8995 hybrid_clock.cc:648] HybridClock initialized: now 1785991205498099 us; error 34 us; skew 500 ppm
Aug 06 04:40:05 dist-test-slave-vkjp krb5kdc[8728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991205, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:05.501119 8995 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:40:05.502118 8995 webserver.cc:460] Webserver started at http://127.31.166.130:38689/ using document root <none> and password file <none>
I20260806 04:40:05.502698 8995 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:05.502764 8995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:05.502992 8995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:05.505177 8995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "3a8f05ba4c6b445ebf9a4966f74f66ec"
format_stamp: "Formatted at 2026-08-06 04:40:05 on dist-test-slave-vkjp"
server_key: "e091c2141a96988b7cd6f30b9c47b7d6"
server_key_iv: "f7fe841676763a605ab9d6d3f2a76c3c"
server_key_version: "encryptionkey@0"
I20260806 04:40:05.505786 8995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "3a8f05ba4c6b445ebf9a4966f74f66ec"
format_stamp: "Formatted at 2026-08-06 04:40:05 on dist-test-slave-vkjp"
server_key: "e091c2141a96988b7cd6f30b9c47b7d6"
server_key_iv: "f7fe841676763a605ab9d6d3f2a76c3c"
server_key_version: "encryptionkey@0"
I20260806 04:40:05.509859 8995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:40:05.512600 9013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:05.513983 8995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:40:05.514153 8995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "3a8f05ba4c6b445ebf9a4966f74f66ec"
format_stamp: "Formatted at 2026-08-06 04:40:05 on dist-test-slave-vkjp"
server_key: "e091c2141a96988b7cd6f30b9c47b7d6"
server_key_iv: "f7fe841676763a605ab9d6d3f2a76c3c"
server_key_version: "encryptionkey@0"
I20260806 04:40:05.514283 8995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:05.531816 8995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:05.535537 8995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:05.535764 8995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:05.536449 8995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:05.537384 8995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:05.537460 8995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:05.537536 8995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:05.537580 8995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:05.547991 8995 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:38291
I20260806 04:40:05.548005 9137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:38291 every 8 connection(s)
I20260806 04:40:05.548967 8995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:40:05.555259 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 8995
I20260806 04:40:05.555424 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:40:05.555747 32410 external_mini_cluster.cc:1428] Setting key cabbe83e30bcb2a156fcd921b66d9dfc
Aug 06 04:40:05 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991205, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:40:05.563838 8822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:05.550771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:40357 (local address 127.31.166.190:40931)
0806 04:40:05.550903 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:05.550906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:05.551848 (+ 942us) server_negotiation.cc:407] Connection header received
0806 04:40:05.553159 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:05.553162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:05.553216 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:05.553307 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:05.554418 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:05.555000 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:05.556576 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:05.556926 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:05.557045 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:05.559942 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:05.559946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:05.559949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:05.559975 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:05.561546 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:05.562121 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:05.562125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:05.562126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:05.562212 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:05.562590 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:05.562594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:05.562596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:05.562795 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:05.562910 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:05.563505 (+ 595us) server_negotiation.cc:299] Negotiation successful
0806 04:40:05.563637 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":42}
I20260806 04:40:05.564649 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:05.551234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:40357)
0806 04:40:05.551690 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:05.551732 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:05.552954 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:05.553473 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:05.553479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:05.553819 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:05.554281 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:05.554293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:05.555150 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:05.555153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:05.556422 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:05.556432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:05.557075 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:05.557078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:05.557285 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:05.557943 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:05.557970 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:05.559753 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:05.561689 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:05.561695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:05.561710 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:05.561996 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:05.562327 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:05.562330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:05.562333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:05.562483 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:05.562907 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:05.562915 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:05.563247 (+ 332us) client_negotiation.cc:769] Sending connection context
0806 04:40:05.563440 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 04:40:05.563801 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:40:05.565902 9138 heartbeater.cc:347] Connected to a master server at 127.31.166.190:40931
I20260806 04:40:05.566139 9138 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:05.566696 9138 heartbeater.cc:511] Master 127.31.166.190:40931 requested a full tablet report, sending...
I20260806 04:40:05.567819 8779 ts_manager.cc:194] Registered new tserver with Master: 3a8f05ba4c6b445ebf9a4966f74f66ec (127.31.166.130:38291)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
I20260806 04:40:05.568563 8779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:40357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:40:05.586542 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:40931
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:40:05.708174 9143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:05.708457 9143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:05.708504 9143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:05.712628 9143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:05.712704 9143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:05.712766 9143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:40:05.718212 9143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:05.719384 9143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:05.720352 9143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:05.730717 9149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:05.730722 9148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:05.730717 9151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:05.730980 9143 server_base.cc:1034] running on GCE node
I20260806 04:40:05.731529 9143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:05.732185 9143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:05.733383 9143 hybrid_clock.cc:648] HybridClock initialized: now 1785991205733357 us; error 41 us; skew 500 ppm
Aug 06 04:40:05 dist-test-slave-vkjp krb5kdc[8728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991205, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:05.736182 9143 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:40:05.737162 9143 webserver.cc:460] Webserver started at http://127.31.166.131:39051/ using document root <none> and password file <none>
I20260806 04:40:05.737721 9143 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:05.737809 9143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:05.738059 9143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:05.740319 9143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "7229adc7f3a94d5aa22577829e234a09"
format_stamp: "Formatted at 2026-08-06 04:40:05 on dist-test-slave-vkjp"
server_key: "98a028dee3450e3b904c8e91371dc076"
server_key_iv: "26fccdff0bb9fea4befee07e9125f1de"
server_key_version: "encryptionkey@0"
I20260806 04:40:05.740892 9143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "7229adc7f3a94d5aa22577829e234a09"
format_stamp: "Formatted at 2026-08-06 04:40:05 on dist-test-slave-vkjp"
server_key: "98a028dee3450e3b904c8e91371dc076"
server_key_iv: "26fccdff0bb9fea4befee07e9125f1de"
server_key_version: "encryptionkey@0"
I20260806 04:40:05.744599 9143 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:40:05.747426 9159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:05.748667 9143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:40:05.748809 9143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "7229adc7f3a94d5aa22577829e234a09"
format_stamp: "Formatted at 2026-08-06 04:40:05 on dist-test-slave-vkjp"
server_key: "98a028dee3450e3b904c8e91371dc076"
server_key_iv: "26fccdff0bb9fea4befee07e9125f1de"
server_key_version: "encryptionkey@0"
I20260806 04:40:05.748941 9143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:05.763783 9143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:05.766927 9143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:05.767179 9143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:05.767774 9143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:05.768704 9143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:05.768772 9143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:05.768839 9143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:05.768893 9143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:05.777882 9143 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:44591
I20260806 04:40:05.777901 9279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:44591 every 8 connection(s)
I20260806 04:40:05.778852 9143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:40:05.785710 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 9143
I20260806 04:40:05.785877 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:40:05.786281 32410 external_mini_cluster.cc:1428] Setting key b28a02f4c96f2411ba66a4bb1d37ea5c
Aug 06 04:40:05 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991205, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:40:05.792882 8822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:05.780739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:36895 (local address 127.31.166.190:40931)
0806 04:40:05.780914 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:05.780919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:05.781755 (+ 836us) server_negotiation.cc:407] Connection header received
0806 04:40:05.783051 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:05.783054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:05.783152 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:05.783239 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:05.784361 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:05.784899 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:05.786088 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:05.786439 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:05.786556 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:05.789441 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:05.789444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:05.789446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:05.789477 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:05.790681 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:05.791250 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:05.791253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:05.791254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:05.791450 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:05.791786 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:05.791789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:05.791790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:05.792042 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:05.792142 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:05.792639 (+ 497us) server_negotiation.cc:299] Negotiation successful
0806 04:40:05.792751 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 04:40:05.793759 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:05.781152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:36895)
0806 04:40:05.781598 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:05.781642 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:05.782839 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:05.783394 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:05.783400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:05.783740 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:05.784225 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:05.784238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:05.785004 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:05.785006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:05.785965 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:05.785981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:05.786597 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:05.786600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:05.786741 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:05.787446 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:05.787469 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:05.789291 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:05.790819 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:05.790824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:05.790836 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:05.791152 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:05.791556 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:05.791558 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:05.791560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:05.791696 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:05.792141 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:05.792146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:05.792386 (+ 240us) client_negotiation.cc:769] Sending connection context
0806 04:40:05.792584 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 04:40:05.792875 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":122,"threads_started":1}
I20260806 04:40:05.794847 9281 heartbeater.cc:347] Connected to a master server at 127.31.166.190:40931
I20260806 04:40:05.795125 9281 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:05.795686 9281 heartbeater.cc:511] Master 127.31.166.190:40931 requested a full tablet report, sending...
I20260806 04:40:05.796711 8779 ts_manager.cc:194] Registered new tserver with Master: 7229adc7f3a94d5aa22577829e234a09 (127.31.166.131:44591)
I20260806 04:40:05.797302 8779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:36895
I20260806 04:40:05.800709 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.129.keytab.
Entry for principal oryx/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.129.keytab.
WARNING: no policy specified for oryx/127.31.166.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.166.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.130.keytab.
Entry for principal oryx/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.130.keytab.
WARNING: no policy specified for oryx/127.31.166.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.166.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.131.keytab.
Entry for principal oryx/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.131.keytab.
I20260806 04:40:05.878280 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 8847
I20260806 04:40:05.886698 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 8995
I20260806 04:40:05.894733 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 9143
I20260806 04:40:05.901198 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 8742
I20260806 04:40:05.908339 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:40931
--webserver_interface=127.31.166.190
--webserver_port=43913
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:40931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:40:06.019634 9290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:06.019939 9290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:06.019986 9290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:06.024466 9290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:40:06.024530 9290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:06.024552 9290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:06.024575 9290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:40:06.024593 9290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:40:06.030313 9290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:40931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:40931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=43913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:06.031379 9290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:06.032373 9290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:06.041003 9299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:06.041029 9297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:06.041025 9296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:06.041081 9290 server_base.cc:1034] running on GCE node
I20260806 04:40:06.041776 9290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:06.042975 9290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:06.044178 9290 hybrid_clock.cc:648] HybridClock initialized: now 1785991206044166 us; error 29 us; skew 500 ppm
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991206, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:06.047250 9290 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:40:06.048308 9290 webserver.cc:460] Webserver started at http://127.31.166.190:43913/ using document root <none> and password file <none>
I20260806 04:40:06.048916 9290 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:06.049039 9290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:06.053144 9290 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:40:06.055848 9306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:06.057045 9290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:40:06.057194 9290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "82f0657b71ce4fb9ba17811490530492"
format_stamp: "Formatted at 2026-08-06 04:40:04 on dist-test-slave-vkjp"
server_key: "445b5c9d36e295d055128c889b67ce30"
server_key_iv: "82cdcb8440dad10902e674f3dc4ac3d1"
server_key_version: "encryptionkey@0"
I20260806 04:40:06.057803 9290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:06.085258 9290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:06.088647 9290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:06.088865 9290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:06.096400 9290 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:40931
I20260806 04:40:06.096427 9362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:40931 every 8 connection(s)
I20260806 04:40:06.097455 9290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:40:06.101768 9363 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:40:06.103514 9363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492: Bootstrap starting.
I20260806 04:40:06.104947 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 9290
I20260806 04:40:06.105504 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:37233
--local_ip_for_outbound_sockets=127.31.166.129
--tserver_master_addrs=127.31.166.190:40931
--webserver_port=34521
--webserver_interface=127.31.166.129
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.166.129
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:40:06.121943 9363 log.cc:824] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492: Log is configured to *not* fsync() on all Append() calls
I20260806 04:40:06.129088 9363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:40:06.129546 9363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492: Bootstrap complete.
I20260806 04:40:06.133981 9363 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f0657b71ce4fb9ba17811490530492" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40931 } }
I20260806 04:40:06.134394 9363 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f0657b71ce4fb9ba17811490530492, State: Initialized, Role: FOLLOWER
I20260806 04:40:06.134944 9363 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f0657b71ce4fb9ba17811490530492" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40931 } }
I20260806 04:40:06.135071 9363 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:40:06.135183 9363 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:40:06.135313 9363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:40:06.137566 9363 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f0657b71ce4fb9ba17811490530492" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40931 } }
I20260806 04:40:06.137931 9363 leader_election.cc:302] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82f0657b71ce4fb9ba17811490530492; no voters:
I20260806 04:40:06.138262 9363 leader_election.cc:288] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 04:40:06.138409 9367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:40:06.138644 9367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [term 2 LEADER]: Becoming Leader. State: Replica: 82f0657b71ce4fb9ba17811490530492, State: Running, Role: LEADER
I20260806 04:40:06.139158 9367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f0657b71ce4fb9ba17811490530492" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40931 } }
I20260806 04:40:06.139622 9363 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:40:06.142561 9368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "82f0657b71ce4fb9ba17811490530492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f0657b71ce4fb9ba17811490530492" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40931 } } }
I20260806 04:40:06.142699 9368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [sys.catalog]: This master's current role is: LEADER
I20260806 04:40:06.143005 9369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82f0657b71ce4fb9ba17811490530492. Latest consensus state: current_term: 2 leader_uuid: "82f0657b71ce4fb9ba17811490530492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82f0657b71ce4fb9ba17811490530492" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40931 } } }
I20260806 04:40:06.143158 9369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492 [sys.catalog]: This master's current role is: LEADER
I20260806 04:40:06.145538 9382 catalog_manager.cc:1259] Loaded cluster ID: d704157390c342fab359fee7ca53ccba
I20260806 04:40:06.145605 9382 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492: loading cluster ID for follower catalog manager: success
I20260806 04:40:06.148253 9382 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492: acquiring CA information for follower catalog manager: success
I20260806 04:40:06.149770 9382 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:40:06.150002 9383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:40:06.151216 9383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:40:06.151657 9383 catalog_manager.cc:1259] Loaded cluster ID: d704157390c342fab359fee7ca53ccba
I20260806 04:40:06.151715 9383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:40:06.152320 9383 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:40:06.152798 9383 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 82f0657b71ce4fb9ba17811490530492: Loaded TSK: 0
I20260806 04:40:06.153509 9383 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:40:06.225414 9365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:06.225661 9365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:06.225704 9365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:06.229794 9365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:06.229861 9365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:06.229948 9365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:40:06.235311 9365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.129.keytab
--principal=oryx/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:37233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=34521
--tserver_master_addrs=127.31.166.190:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:06.236344 9365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:06.237512 9365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:06.247140 9389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:06.247128 9390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:06.247409 9392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:06.247612 9365 server_base.cc:1034] running on GCE node
I20260806 04:40:06.248037 9365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:06.248790 9365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:06.249979 9365 hybrid_clock.cc:648] HybridClock initialized: now 1785991206249948 us; error 47 us; skew 500 ppm
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991206, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:06.253453 9365 init.cc:377] Logged in from keytab as oryx/127.31.166.129@KRBTEST.COM (short username oryx)
I20260806 04:40:06.254487 9365 webserver.cc:460] Webserver started at http://127.31.166.129:34521/ using document root <none> and password file <none>
I20260806 04:40:06.255193 9365 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:06.255318 9365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:06.260740 9365 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:40:06.263371 9400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:06.264868 9365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:40:06.265009 9365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "c2d0e9edf9da4a708307eba3022dc64e"
format_stamp: "Formatted at 2026-08-06 04:40:05 on dist-test-slave-vkjp"
server_key: "74d5aaa37fc527cec951e4124550093a"
server_key_iv: "adb6048091b272691ef8ad94b84222b1"
server_key_version: "encryptionkey@0"
I20260806 04:40:06.265738 9365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:06.290947 9365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:06.294513 9365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:06.294740 9365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:06.295435 9365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:06.296344 9365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:06.296418 9365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:06.296496 9365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:06.296540 9365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:06.307054 9365 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:37233
I20260806 04:40:06.307099 9522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:37233 every 8 connection(s)
I20260806 04:40:06.308133 9365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:40:06.311352 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 9365
I20260806 04:40:06.311798 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:38291
--local_ip_for_outbound_sockets=127.31.166.130
--tserver_master_addrs=127.31.166.190:40931
--webserver_port=38689
--webserver_interface=127.31.166.130
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.166.130
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:06.322993 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:06.310404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:51657)
0806 04:40:06.310902 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:06.310958 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:06.312495 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:06.313703 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:06.313710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:06.314130 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:06.314568 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.314581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.316487 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.316489 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.317357 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.317367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.318512 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.318515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.318651 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.319305 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:06.319321 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:06.322136 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":149,"threads_started":1}
W20260806 04:40:06.323495 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:06.310203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:51657 (local address 127.31.166.190:40931)
0806 04:40:06.310649 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:06.310668 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:06.311072 (+ 404us) server_negotiation.cc:407] Connection header received
0806 04:40:06.312925 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:06.312942 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:06.313298 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:06.313631 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:06.314749 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.316343 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.317499 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.318318 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.318527 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.322472 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:51657: BlockingRecv error: recv got EOF from 127.31.166.129:51657 (error 108)
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":144,"threads_started":1}
W20260806 04:40:06.323832 9523 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:40931 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.166.190:40931: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:06.329983 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:06.324380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:45125)
0806 04:40:06.324560 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:06.324574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:06.324658 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:06.324937 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:06.324940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:06.325112 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:06.325227 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.325234 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.325977 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.325980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.326882 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.326891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.327397 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.327400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.327506 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.327966 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:06.327981 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:06.329838 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113}
W20260806 04:40:06.330075 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:06.324490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:45125 (local address 127.31.166.190:40931)
0806 04:40:06.324606 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:06.324609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:06.324622 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:40:06.324790 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:06.324792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:06.324836 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:06.324899 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:06.325349 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.325877 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.327009 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.327292 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.327375 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.329963 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:45125: BlockingRecv error: recv got EOF from 127.31.166.129:45125 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:06.335817 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:06.330697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:55911)
0806 04:40:06.330800 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:06.330813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:06.330880 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:06.331135 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:06.331142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:06.331259 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:06.331335 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.331340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.332332 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.332335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.332838 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.332845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.333404 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.333406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.333463 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.333932 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:06.333944 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:06.335711 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260806 04:40:06.335976 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:06.330782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:55911 (local address 127.31.166.190:40931)
0806 04:40:06.330893 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:06.330897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:06.330910 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:40:06.330945 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:06.330948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:06.330988 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:06.331071 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:06.331443 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.332215 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.332945 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.333285 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.333379 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.335863 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:55911: BlockingRecv error: recv got EOF from 127.31.166.129:55911 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 04:40:06.336077 9523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:40:06.428042 9527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:06.428354 9527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:06.428398 9527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:06.432492 9527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:06.432555 9527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:06.432641 9527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:40:06.437938 9527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.130.keytab
--principal=oryx/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:38291
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=38689
--tserver_master_addrs=127.31.166.190:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:06.438969 9527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:06.440033 9527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:06.449426 9534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:06.449501 9533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:06.449548 9536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:06.449808 9527 server_base.cc:1034] running on GCE node
I20260806 04:40:06.450196 9527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:06.450762 9527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:06.451942 9527 hybrid_clock.cc:648] HybridClock initialized: now 1785991206451901 us; error 58 us; skew 500 ppm
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991206, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:06.454517 9527 init.cc:377] Logged in from keytab as oryx/127.31.166.130@KRBTEST.COM (short username oryx)
I20260806 04:40:06.455343 9527 webserver.cc:460] Webserver started at http://127.31.166.130:38689/ using document root <none> and password file <none>
I20260806 04:40:06.455833 9527 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:06.455910 9527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:06.459558 9527 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:40:06.461597 9543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:06.462914 9527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:40:06.463016 9527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "3a8f05ba4c6b445ebf9a4966f74f66ec"
format_stamp: "Formatted at 2026-08-06 04:40:05 on dist-test-slave-vkjp"
server_key: "e091c2141a96988b7cd6f30b9c47b7d6"
server_key_iv: "f7fe841676763a605ab9d6d3f2a76c3c"
server_key_version: "encryptionkey@0"
I20260806 04:40:06.463564 9527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:06.479015 9527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:06.481956 9527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:06.482124 9527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:06.482653 9527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:06.483451 9527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:06.483498 9527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:06.483537 9527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:06.483552 9527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:06.492766 9527 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:38291
I20260806 04:40:06.492847 9660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:38291 every 8 connection(s)
I20260806 04:40:06.493768 9527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:40:06.499405 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 9527
I20260806 04:40:06.499838 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:44591
--local_ip_for_outbound_sockets=127.31.166.131
--tserver_master_addrs=127.31.166.190:40931
--webserver_port=39051
--webserver_interface=127.31.166.131
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:41569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.166.131
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 04:40:06.505833 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:06.495642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:39183 (local address 127.31.166.190:40931)
0806 04:40:06.495819 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:06.495823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:06.496583 (+ 760us) server_negotiation.cc:407] Connection header received
0806 04:40:06.497964 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:06.497967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:06.498029 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:06.498133 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:06.499343 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.500044 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.501729 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.502112 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.502229 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.505662 (+ 3433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:39183: BlockingRecv error: recv got EOF from 127.31.166.130:39183 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 04:40:06.506225 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:06.496008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:39183)
0806 04:40:06.496434 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:06.496473 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:06.497751 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:06.498303 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:06.498310 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:06.498639 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:06.499150 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.499172 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.500176 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.500183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.501605 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.501615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.502256 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.502259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.502391 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.502971 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:06.502987 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:06.505465 (+ 2478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":112,"threads_started":1}
W20260806 04:40:06.506788 9661 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:40931 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.166.190:40931: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:06.512249 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:06.507313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:42243)
0806 04:40:06.507421 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:06.507436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:06.507511 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:06.507723 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:06.507726 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:06.507894 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:06.507977 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.507983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.508670 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.508675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.509203 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.509210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.509649 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.509653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.509743 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.510209 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:06.510220 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:06.512116 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 04:40:06.512387 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:06.507382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:42243 (local address 127.31.166.190:40931)
0806 04:40:06.507520 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:06.507524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:06.507538 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:40:06.507582 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:06.507585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:06.507623 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:06.507709 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:06.508083 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.508569 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.509314 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.509538 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.509636 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.512229 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:42243: BlockingRecv error: recv got EOF from 127.31.166.130:42243 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:06.518162 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:06.512942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:60617)
0806 04:40:06.513044 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:06.513062 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:06.513118 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:06.513341 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:06.513343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:06.513455 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:06.513523 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.513527 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.514497 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.514500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.515054 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.515062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.515674 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.515677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.515738 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.516209 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:06.516222 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:06.518049 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260806 04:40:06.518352 9661 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:40:06.518486 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:06.513013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:60617 (local address 127.31.166.190:40931)
0806 04:40:06.513118 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:06.513122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:06.513136 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:40:06.513172 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:06.513174 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:06.513211 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:06.513293 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:06.513618 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.514377 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.515224 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.515563 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.515653 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.518189 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:60617: BlockingRecv error: recv got EOF from 127.31.166.130:60617 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 04:40:06.612377 9665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:06.612641 9665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:06.612684 9665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:06.616597 9665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:06.616649 9665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:06.616707 9665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:40:06.621949 9665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:41569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/krb5kdc/oryx_127.31.166.131.keytab
--principal=oryx/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:44591
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=39051
--tserver_master_addrs=127.31.166.190:40931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:06.622975 9665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:06.624097 9665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:06.633795 9671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:06.633824 9673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:06.633807 9670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:06.634088 9665 server_base.cc:1034] running on GCE node
I20260806 04:40:06.634564 9665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:06.635290 9665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:06.636493 9665 hybrid_clock.cc:648] HybridClock initialized: now 1785991206636481 us; error 27 us; skew 500 ppm
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991206, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:06.639626 9665 init.cc:377] Logged in from keytab as oryx/127.31.166.131@KRBTEST.COM (short username oryx)
I20260806 04:40:06.640673 9665 webserver.cc:460] Webserver started at http://127.31.166.131:39051/ using document root <none> and password file <none>
I20260806 04:40:06.641331 9665 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:06.641444 9665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:06.645552 9665 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:40:06.647995 9680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:06.649336 9665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:40:06.649477 9665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "7229adc7f3a94d5aa22577829e234a09"
format_stamp: "Formatted at 2026-08-06 04:40:05 on dist-test-slave-vkjp"
server_key: "98a028dee3450e3b904c8e91371dc076"
server_key_iv: "26fccdff0bb9fea4befee07e9125f1de"
server_key_version: "encryptionkey@0"
I20260806 04:40:06.650019 9665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:06.677915 9665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:06.681563 9665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:06.681787 9665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:06.682425 9665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:06.683358 9665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:06.683430 9665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:06.683518 9665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:06.683568 9665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:06.693225 9665 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:44591
I20260806 04:40:06.693246 9803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:44591 every 8 connection(s)
I20260806 04:40:06.694386 9665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:40:06.696913 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 9665
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 04:40:06.706823 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:06.696338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:56737 (local address 127.31.166.190:40931)
0806 04:40:06.696519 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:06.696524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:06.697450 (+ 926us) server_negotiation.cc:407] Connection header received
0806 04:40:06.699262 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:06.699266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:06.699314 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:06.699383 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:06.700607 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.701240 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.702611 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.702858 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.702976 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.706649 (+ 3673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:56737: BlockingRecv error: recv got EOF from 127.31.166.131:56737 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 04:40:06.707386 9808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:06.698062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51876 (local address 127.31.166.190:40931)
0806 04:40:06.698350 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:06.698355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:06.698374 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:40:06.698432 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:06.698436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:06.698486 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:06.698590 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:06.699383 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.699939 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.700816 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.701059 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.701167 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.702764 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:06.702771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:06.702785 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:06.702815 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:06.705075 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:06.705554 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:06.705561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:06.705567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:06.705738 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:06.706050 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:06.706055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:06.706057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:06.706435 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:06.706655 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:06.706970 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 04:40:06.707251 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
I20260806 04:40:06.707896 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:06.696728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:56737)
0806 04:40:06.697262 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:06.697338 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:06.699013 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:06.699554 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:06.699561 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:06.699915 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:06.700462 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.700475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.701375 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.701379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.702477 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.702487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.703022 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.703027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.703248 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.703799 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:06.703817 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:06.706537 (+ 2720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":122,"threads_started":1}
W20260806 04:40:06.708460 9804 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:40931 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.166.190:40931: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:06.714653 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:06.708907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:35899)
0806 04:40:06.709048 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:06.709065 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:06.709148 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:06.709560 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:06.709563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:06.709712 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:06.709858 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.709866 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.710820 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.710823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.711462 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.711472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.711963 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.711966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.712031 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.712532 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:06.712544 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:06.714476 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 04:40:06.714742 9808 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:06.709013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:35899 (local address 127.31.166.190:40931)
0806 04:40:06.709126 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:06.709129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:06.709141 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:40:06.709253 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:06.709257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:06.709361 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:06.709441 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:06.709998 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.710706 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.711588 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.711859 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.711960 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.714616 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:35899: BlockingRecv error: recv got EOF from 127.31.166.131:35899 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:06 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:06.720381 9806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:06.715419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:32855)
0806 04:40:06.715536 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:06.715554 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:06.715643 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:06.715905 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:06.715907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:06.716044 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:06.716123 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.716128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.716823 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.716826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.717403 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:06.717411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.717837 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.717841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:06.717907 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.718355 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:06.718368 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:06.720234 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 04:40:06.720454 9808 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:06.715426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:32855 (local address 127.31.166.190:40931)
0806 04:40:06.715567 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:06.715570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:06.715617 (+ 47us) server_negotiation.cc:407] Connection header received
0806 04:40:06.715728 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:06.715730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:06.715767 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:06.715827 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:06.716229 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.716719 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.717511 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:06.717707 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:06.717776 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:06.720357 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:32855: BlockingRecv error: recv got EOF from 127.31.166.131:32855 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260806 04:40:06.720762 9804 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 04:40:07 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:07 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:07.342571 9809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:07.336708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:35179)
0806 04:40:07.336979 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:07.336993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:07.337091 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:07.337404 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:07.337407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:07.337599 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:07.337713 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:07.337719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:07.338460 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:07.338463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:07.339112 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:07.339120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:07.339557 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:07.339578 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:07.339656 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:07.340406 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:07.340418 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:07.342443 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":90,"threads_started":1}
W20260806 04:40:07.342746 9810 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:07.336781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:35179 (local address 127.31.166.190:40931)
0806 04:40:07.337031 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:07.337035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:07.337048 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:40:07.337180 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:07.337182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:07.337229 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:07.337307 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:07.337818 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:07.338351 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:07.339208 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:07.339439 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:07.339522 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:07.342578 (+ 3056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:35179: BlockingRecv error: recv got EOF from 127.31.166.129:35179 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":72,"threads_started":1}
Aug 06 04:40:07 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:07 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:07.525074 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:07.519037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:40007)
0806 04:40:07.519392 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:07.519414 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:07.519561 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:07.519941 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:07.519944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:07.520187 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:07.520340 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:07.520349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:07.521256 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:07.521259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:07.521917 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:07.521926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:07.522337 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:07.522339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:07.522407 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:07.522964 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:07.522977 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:07.524948 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":143,"threads_started":1}
W20260806 04:40:07.525204 9810 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:07.519212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:40007 (local address 127.31.166.190:40931)
0806 04:40:07.519435 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:07.519442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:07.519506 (+ 64us) server_negotiation.cc:407] Connection header received
0806 04:40:07.519730 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:07.519734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:07.519795 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:07.519918 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:07.520482 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:07.521125 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:07.522022 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:07.522232 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:07.522305 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:07.525084 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:40007: BlockingRecv error: recv got EOF from 127.31.166.130:40007 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 06 04:40:07 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:07 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:07.727357 9812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:07.721632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:57207)
0806 04:40:07.721944 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:07.721957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:07.722099 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:07.722399 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:07.722402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:07.722639 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:07.722766 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:07.722773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:07.723651 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:07.723654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:07.724240 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:07.724247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:07.724758 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:07.724761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:07.724821 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:07.725372 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:07.725388 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:07.727220 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
W20260806 04:40:07.727612 9810 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:07.721685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:57207 (local address 127.31.166.190:40931)
0806 04:40:07.721824 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:07.721829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:07.722066 (+ 237us) server_negotiation.cc:407] Connection header received
0806 04:40:07.722204 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:07.722210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:07.722275 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:07.722344 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:07.722892 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:07.723541 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:07.724391 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:07.724642 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:07.724737 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:07.727474 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:57207: BlockingRecv error: recv got EOF from 127.31.166.131:57207 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 04:40:08 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:08 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:08.348518 9813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:08.343333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:50001)
0806 04:40:08.343648 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:08.343662 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:08.343755 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:08.344003 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:08.344006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:08.344180 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:08.344275 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:08.344280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:08.345033 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:08.345036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:08.345593 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:08.345601 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:08.345995 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:08.345998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:08.346058 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:08.346537 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:08.346548 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:08.348371 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":88,"threads_started":1}
W20260806 04:40:08.348696 9814 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:08.343459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:50001 (local address 127.31.166.190:40931)
0806 04:40:08.343698 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:08.343701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:08.343714 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:40:08.343839 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:08.343841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:08.343890 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:08.343967 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:08.344392 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:08.344936 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:08.345687 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:08.345903 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:08.345977 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:08.348526 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:50001: BlockingRecv error: recv got EOF from 127.31.166.129:50001 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":95,"threads_started":1}
Aug 06 04:40:08 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:08 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:08.531649 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:08.526037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:44303)
0806 04:40:08.526318 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:08.526332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:08.526463 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:08.526716 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:08.526718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:08.526903 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:08.527004 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:08.527009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:08.527839 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:08.527841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:08.528425 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:08.528434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:08.528874 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:08.528877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:08.528941 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:08.529514 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:08.529526 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:08.531472 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
W20260806 04:40:08.531700 9814 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:08.526159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:44303 (local address 127.31.166.190:40931)
0806 04:40:08.526312 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:08.526318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:08.526430 (+ 112us) server_negotiation.cc:407] Connection header received
0806 04:40:08.526554 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:08.526556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:08.526607 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:08.526683 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:08.527163 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:08.527723 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:08.528545 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:08.528757 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:08.528859 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:08.531594 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:44303: BlockingRecv error: recv got EOF from 127.31.166.130:44303 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 04:40:08 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:08 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:08.737694 9816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:08.728191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:49977)
0806 04:40:08.728570 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:08.728587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:08.728747 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:08.729052 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:08.729055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:08.729271 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:08.729376 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:08.729382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:08.730141 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:08.730146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:08.730747 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:08.730755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:08.731282 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:08.731286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:08.731364 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:08.735412 (+ 4048us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:08.735426 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:08.737566 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":133,"threads_started":1}
W20260806 04:40:08.737862 9814 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:08.728399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:49977 (local address 127.31.166.190:40931)
0806 04:40:08.728571 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:08.728577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:08.728683 (+ 106us) server_negotiation.cc:407] Connection header received
0806 04:40:08.728887 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:08.728890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:08.728940 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:08.729026 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:08.729504 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:08.730031 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:08.730877 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:08.731162 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:08.731265 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:08.737726 (+ 6461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:49977: BlockingRecv error: recv got EOF from 127.31.166.131:49977 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 06 04:40:09 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:09 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:09.355464 9818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:09.349426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:36865)
0806 04:40:09.349775 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:09.349792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:09.349898 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:09.350166 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:09.350169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:09.350395 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:09.350514 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:09.350522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:09.351393 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:09.351396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:09.352004 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:09.352014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:09.352469 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:09.352474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:09.352560 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:09.353291 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:09.353309 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:09.355335 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":126,"threads_started":1}
W20260806 04:40:09.355618 9819 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:09.349574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:36865 (local address 127.31.166.190:40931)
0806 04:40:09.349940 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:09.349944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:09.349956 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:40:09.349990 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:09.349992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:09.350033 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:09.350113 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:09.350675 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:09.351268 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:09.352111 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:09.352335 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:09.352419 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:09.355486 (+ 3067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:36865: BlockingRecv error: recv got EOF from 127.31.166.129:36865 (error 108)
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":210,"threads_started":1}
Aug 06 04:40:09 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:09 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:09.537865 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:09.532516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:53927)
0806 04:40:09.532840 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:09.532859 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:09.532972 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:09.533241 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:09.533244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:09.533491 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:09.533589 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:09.533595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:09.534348 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:09.534351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:09.534900 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:09.534907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:09.535387 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:09.535391 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:09.535479 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:09.535936 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:09.535947 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:09.537710 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":97,"threads_started":1}
W20260806 04:40:09.538024 9819 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:09.532597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:53927 (local address 127.31.166.190:40931)
0806 04:40:09.532738 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:09.532742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:09.532941 (+ 199us) server_negotiation.cc:407] Connection header received
0806 04:40:09.533066 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:09.533069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:09.533124 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:09.533213 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:09.533697 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:09.534216 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:09.535002 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:09.535270 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:09.535376 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:09.537888 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:53927: BlockingRecv error: recv got EOF from 127.31.166.130:53927 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 06 04:40:09 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:09 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:09.744320 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:09.738522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:53939)
0806 04:40:09.738857 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:09.738870 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:09.738968 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:09.739215 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:09.739217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:09.739439 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:09.739536 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:09.739541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:09.740400 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:09.740403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:09.741213 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:09.741224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:09.741766 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:09.741769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:09.741835 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:09.742327 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:09.742339 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:09.744141 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":90,"threads_started":1}
W20260806 04:40:09.744441 9819 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:09.738638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:53939 (local address 127.31.166.190:40931)
0806 04:40:09.738772 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:09.738776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:09.738937 (+ 161us) server_negotiation.cc:407] Connection header received
0806 04:40:09.739043 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:09.739045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:09.739113 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:09.739185 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:09.739685 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:09.740207 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:09.741340 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:09.741662 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:09.741754 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:09.744315 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:53939: BlockingRecv error: recv got EOF from 127.31.166.131:53939 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 06 04:40:10 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:10 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:10.361807 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:10.356377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:44247)
0806 04:40:10.356735 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:10.356749 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:10.356849 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:10.357097 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:10.357100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:10.357281 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:10.357407 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:10.357413 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:10.358141 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:10.358143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:10.358701 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:10.358709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:10.359196 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:10.359198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:10.359263 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:10.359756 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:10.359768 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:10.361678 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":92,"threads_started":1}
W20260806 04:40:10.362002 9823 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:10.356440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:44247 (local address 127.31.166.190:40931)
0806 04:40:10.356690 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:10.356696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:10.356836 (+ 140us) server_negotiation.cc:407] Connection header received
0806 04:40:10.356933 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:10.356935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:10.356982 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:10.357059 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:10.357509 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:10.358051 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:10.358800 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:10.359034 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:10.359143 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:10.361832 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:44247: BlockingRecv error: recv got EOF from 127.31.166.129:44247 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":63,"threads_started":1}
Aug 06 04:40:10 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:10 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:10.544366 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:10.538665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:60405)
0806 04:40:10.539000 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:10.539013 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:10.539159 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:10.539388 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:10.539390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:10.539619 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:10.539720 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:10.539726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:10.540466 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:10.540469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:10.541094 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:10.541103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:10.541601 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:10.541604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:10.541665 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:10.542201 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:10.542213 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:10.544199 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":102,"threads_started":1}
W20260806 04:40:10.544442 9823 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:10.538786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:60405 (local address 127.31.166.190:40931)
0806 04:40:10.538947 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:10.538951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:10.539093 (+ 142us) server_negotiation.cc:407] Connection header received
0806 04:40:10.539235 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:10.539237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:10.539281 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:10.539350 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:10.539820 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:10.540354 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:10.541229 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:10.541488 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:10.541602 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:10.544349 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:60405: BlockingRecv error: recv got EOF from 127.31.166.130:60405 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 04:40:10 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:10 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:10.750590 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:10.745216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:42455)
0806 04:40:10.745504 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:10.745518 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:10.745621 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:10.745843 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:10.745845 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:10.746025 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:10.746125 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:10.746130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:10.746892 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:10.746894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:10.747516 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:10.747525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:10.748023 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:10.748025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:10.748087 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:10.748637 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:10.748648 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:10.750454 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":91,"threads_started":1}
W20260806 04:40:10.750716 9823 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:10.745286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:42455 (local address 127.31.166.190:40931)
0806 04:40:10.745443 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:10.745447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:10.745590 (+ 143us) server_negotiation.cc:407] Connection header received
0806 04:40:10.745687 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:10.745690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:10.745738 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:10.745812 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:10.746224 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:10.746784 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:10.747649 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:10.747925 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:10.748007 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:10.750604 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:42455: BlockingRecv error: recv got EOF from 127.31.166.131:42455 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 04:40:11 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:11 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:11.368369 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:11.362608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:58003)
0806 04:40:11.362921 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:11.362936 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:11.363032 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:11.363315 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:11.363318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:11.363519 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:11.363619 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:11.363625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:11.364355 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:11.364358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:11.365050 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:11.365059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:11.365502 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:11.365505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:11.365573 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:11.366113 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:11.366134 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:11.368214 (+ 2080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":132,"threads_started":1}
W20260806 04:40:11.368494 9827 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:11.362677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:58003 (local address 127.31.166.190:40931)
0806 04:40:11.362927 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:11.362930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:11.362987 (+ 57us) server_negotiation.cc:407] Connection header received
0806 04:40:11.363139 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:11.363142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:11.363186 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:11.363262 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:11.363726 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:11.364250 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:11.365169 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:11.365403 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:11.365479 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:11.368339 (+ 2860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:58003: BlockingRecv error: recv got EOF from 127.31.166.129:58003 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":96,"threads_started":1}
Aug 06 04:40:11 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:11 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:11.551036 9828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:11.545209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:49811)
0806 04:40:11.545488 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:11.545502 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:11.545587 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:11.545846 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:11.545848 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:11.546036 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:11.546132 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:11.546137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:11.546942 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:11.546944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:11.547632 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:11.547641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:11.548099 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:11.548102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:11.548166 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:11.548759 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:11.548774 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:11.550920 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":102,"threads_started":1}
W20260806 04:40:11.551262 9827 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:11.545418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:49811 (local address 127.31.166.190:40931)
0806 04:40:11.545569 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:11.545573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:11.545589 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:40:11.545658 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:11.545660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:11.545709 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:11.545808 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:11.546257 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:11.546830 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:11.547758 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:11.547991 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:11.548090 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:11.551124 (+ 3034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:49811: BlockingRecv error: recv got EOF from 127.31.166.130:49811 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 06 04:40:11 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:11 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:11.757700 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:11.751611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:56093)
0806 04:40:11.751914 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:11.751931 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:11.752067 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:11.752426 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:11.752429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:11.752711 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:11.752853 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:11.752866 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:11.753748 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:11.753751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:11.754440 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:11.754450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:11.754917 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:11.754920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:11.755013 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:11.755660 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:11.755682 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:11.757581 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":120,"threads_started":1}
W20260806 04:40:11.757840 9827 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:11.751650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:56093 (local address 127.31.166.190:40931)
0806 04:40:11.751789 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:11.751794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:11.752026 (+ 232us) server_negotiation.cc:407] Connection header received
0806 04:40:11.752176 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:11.752180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:11.752238 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:11.752377 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:11.753046 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:11.753621 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:11.754555 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:11.754788 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:11.754870 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:11.757720 (+ 2850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:56093: BlockingRecv error: recv got EOF from 127.31.166.131:56093 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 04:40:12 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:12 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:12.374301 9830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:12.369098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:38313)
0806 04:40:12.369390 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:12.369404 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:12.369511 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:12.369746 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:12.369749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:12.369944 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:12.370045 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:12.370050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:12.370784 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:12.370787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:12.371381 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:12.371391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:12.371808 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:12.371811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:12.371877 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:12.372453 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:12.372465 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:12.374184 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":98,"threads_started":1}
W20260806 04:40:12.374553 9831 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:12.369167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:38313 (local address 127.31.166.190:40931)
0806 04:40:12.369395 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:12.369399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:12.369480 (+ 81us) server_negotiation.cc:407] Connection header received
0806 04:40:12.369587 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:12.369590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:12.369635 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:12.369712 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:12.370145 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:12.370685 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:12.371492 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:12.371712 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:12.371789 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:12.374403 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:38313: BlockingRecv error: recv got EOF from 127.31.166.129:38313 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
Aug 06 04:40:12 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:12 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:12.557381 9832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:12.551785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:33957)
0806 04:40:12.552152 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:12.552171 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:12.552301 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:12.552573 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:12.552576 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:12.552887 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:12.552991 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:12.552998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:12.553759 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:12.553765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:12.554411 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:12.554420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:12.554870 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:12.554873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:12.554961 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:12.555496 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:12.555507 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:12.557267 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":132,"threads_started":1}
W20260806 04:40:12.557587 9831 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:12.551929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:33957 (local address 127.31.166.190:40931)
0806 04:40:12.552110 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:12.552115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:12.552242 (+ 127us) server_negotiation.cc:407] Connection header received
0806 04:40:12.552419 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:12.552422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:12.552472 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:12.552552 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:12.553107 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:12.553651 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:12.554530 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:12.554757 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:12.554863 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:12.557452 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:33957: BlockingRecv error: recv got EOF from 127.31.166.130:33957 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 04:40:12 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:12 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:12.764485 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:12.758510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:39231)
0806 04:40:12.758933 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:12.758959 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:12.759152 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:12.759524 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:12.759527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:12.759757 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:12.759858 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:12.759864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:12.760681 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:12.760687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:12.761315 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:12.761324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:12.761850 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:12.761853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:12.761948 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:12.762493 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:12.762506 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:12.764372 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":102,"threads_started":1}
W20260806 04:40:12.764686 9831 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:12.758605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:39231 (local address 127.31.166.190:40931)
0806 04:40:12.758907 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:12.758913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:12.759045 (+ 132us) server_negotiation.cc:407] Connection header received
0806 04:40:12.759282 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:12.759285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:12.759382 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:12.759485 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:12.759984 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:12.760562 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:12.761457 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:12.761705 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:12.761843 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:12.764513 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:39231: BlockingRecv error: recv got EOF from 127.31.166.131:39231 (error 108)
Metrics: {"server-negotiator.queue_time_us":199}
Aug 06 04:40:13 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:13 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:13.380919 9834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:13.375188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:48431)
0806 04:40:13.375538 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:13.375555 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:13.375705 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:13.376012 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:13.376015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:13.376234 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:13.376327 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:13.376333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:13.377115 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:13.377120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:13.377683 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:13.377691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:13.378158 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:13.378161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:13.378250 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:13.378855 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:13.378867 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:13.380803 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":88,"threads_started":1}
W20260806 04:40:13.381114 9835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:13.375284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:48431 (local address 127.31.166.190:40931)
0806 04:40:13.375532 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:13.375536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:13.375651 (+ 115us) server_negotiation.cc:407] Connection header received
0806 04:40:13.375824 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:13.375829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:13.375884 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:13.375988 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:13.376448 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:13.376999 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:13.377811 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:13.378041 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:13.378145 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:13.380956 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:48431: BlockingRecv error: recv got EOF from 127.31.166.129:48431 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":68,"threads_started":1}
Aug 06 04:40:13 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:13 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:13.564109 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:13.558166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:60323)
0806 04:40:13.558579 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:13.558593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:13.558742 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:13.559033 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:13.559036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:13.559289 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:13.559395 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:13.559402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:13.560183 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:13.560188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:13.560745 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:13.560753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:13.561250 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:13.561253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:13.561352 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:13.561984 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:13.561996 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:13.563992 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":92,"threads_started":1}
W20260806 04:40:13.564280 9835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:13.558289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:60323 (local address 127.31.166.190:40931)
0806 04:40:13.558515 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:13.558520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:13.558677 (+ 157us) server_negotiation.cc:407] Connection header received
0806 04:40:13.558864 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:13.558867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:13.558919 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:13.559005 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:13.559526 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:13.560067 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:13.560874 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:13.561128 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:13.561243 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:13.564155 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:60323: BlockingRecv error: recv got EOF from 127.31.166.130:60323 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 06 04:40:13 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:13 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:13.775519 9838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:13.765198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:42677)
0806 04:40:13.765540 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:13.765555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:13.765674 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:13.765919 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:13.765922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:13.766158 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:13.766269 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:13.766277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:13.767044 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:13.767046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:13.767771 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:13.767781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:13.768230 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:13.768233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:13.768305 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:13.773344 (+ 5039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:13.773359 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:13.775395 (+ 2036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":70,"threads_started":1}
W20260806 04:40:13.775744 9835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:13.765357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:42677 (local address 127.31.166.190:40931)
0806 04:40:13.765522 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:13.765526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:13.765624 (+ 98us) server_negotiation.cc:407] Connection header received
0806 04:40:13.765755 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:13.765757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:13.765804 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:13.765891 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:13.766387 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:13.766944 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:13.767918 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:13.768127 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:13.768206 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:13.775614 (+ 7408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:42677: BlockingRecv error: recv got EOF from 127.31.166.131:42677 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 04:40:14 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:14 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:14.387666 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:14.381810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:59799)
0806 04:40:14.382106 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:14.382120 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:14.382217 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:14.382491 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:14.382493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:14.382679 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:14.382776 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:14.382781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:14.383643 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:14.383646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:14.384195 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:14.384203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:14.384797 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:14.384800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:14.384871 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:14.385511 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:14.385527 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:14.387483 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":111,"threads_started":1}
W20260806 04:40:14.387733 9840 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:14.381950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:59799 (local address 127.31.166.190:40931)
0806 04:40:14.382199 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:14.382203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:14.382216 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:40:14.382306 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:14.382308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:14.382353 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:14.382452 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:14.382902 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:14.383538 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:14.384302 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:14.384588 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:14.384719 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:14.387609 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:59799: BlockingRecv error: recv got EOF from 127.31.166.129:59799 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
Aug 06 04:40:14 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:14 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:14.570647 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:14.564909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:58097)
0806 04:40:14.565189 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:14.565203 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:14.565307 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:14.565568 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:14.565570 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:14.565753 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:14.565871 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:14.565877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:14.566829 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:14.566832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:14.567520 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:14.567530 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:14.567948 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:14.567951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:14.568013 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:14.568634 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:14.568646 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:14.570527 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":99,"threads_started":1}
W20260806 04:40:14.570928 9840 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:14.564999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:58097 (local address 127.31.166.190:40931)
0806 04:40:14.565128 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:14.565132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:14.565269 (+ 137us) server_negotiation.cc:407] Connection header received
0806 04:40:14.565409 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:14.565412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:14.565456 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:14.565525 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:14.565969 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:14.566708 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:14.567623 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:14.567842 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:14.567930 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:14.570665 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:58097: BlockingRecv error: recv got EOF from 127.31.166.130:58097 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 06 04:40:14 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:14 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:14.782423 9842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:14.776230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:50481)
0806 04:40:14.776619 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:14.776636 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:14.776776 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:14.777103 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:14.777106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:14.777322 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:14.777422 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:14.777427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:14.778257 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:14.778260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:14.778860 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:14.778868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:14.779383 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:14.779387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:14.779460 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:14.780031 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:14.780043 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:14.782216 (+ 2173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":84,"threads_started":1}
W20260806 04:40:14.782532 9840 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:14.776407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:50481 (local address 127.31.166.190:40931)
0806 04:40:14.776581 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:14.776585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:14.776726 (+ 141us) server_negotiation.cc:407] Connection header received
0806 04:40:14.776898 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:14.776902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:14.776970 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:14.777059 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:14.777551 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:14.778149 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:14.778992 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:14.779244 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:14.779330 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:14.782379 (+ 3049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:50481: BlockingRecv error: recv got EOF from 127.31.166.131:50481 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 04:40:15 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:15 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:15.394773 9843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:15.388655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:39587)
0806 04:40:15.389002 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:15.389020 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:15.389135 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:15.389521 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:15.389525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:15.389753 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:15.389904 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:15.389914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:15.390850 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:15.390853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:15.391444 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:15.391453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:15.391956 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:15.391960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:15.392028 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:15.392540 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:15.392552 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:15.394645 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":146,"threads_started":1}
W20260806 04:40:15.394953 9844 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:15.388746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:39587 (local address 127.31.166.190:40931)
0806 04:40:15.389052 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:15.389056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:15.389073 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:40:15.389244 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:15.389248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:15.389365 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:15.389470 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:15.390066 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:15.390733 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:15.391561 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:15.391791 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:15.391910 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:15.394810 (+ 2900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:39587: BlockingRecv error: recv got EOF from 127.31.166.129:39587 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":77,"threads_started":1}
Aug 06 04:40:15 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:15 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:15.577653 9845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:15.571668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:40661)
0806 04:40:15.571996 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:15.572009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:15.572145 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:15.572479 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:15.572482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:15.572701 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:15.572813 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:15.572819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:15.573578 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:15.573583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:15.574158 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:15.574166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:15.574633 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:15.574636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:15.574724 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:15.575268 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:15.575283 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:15.577525 (+ 2242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
W20260806 04:40:15.577841 9844 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:15.571766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:40661 (local address 127.31.166.190:40931)
0806 04:40:15.571940 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:15.571945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:15.572082 (+ 137us) server_negotiation.cc:407] Connection header received
0806 04:40:15.572279 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:15.572281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:15.572362 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:15.572455 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:15.572939 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:15.573468 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:15.574278 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:15.574513 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:15.574617 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:15.577703 (+ 3086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:40661: BlockingRecv error: recv got EOF from 127.31.166.130:40661 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 04:40:15 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:15 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:15.789677 9846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:15.783331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:39541)
0806 04:40:15.783683 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:15.783704 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:15.783860 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:15.784145 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:15.784148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:15.784375 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:15.784471 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:15.784476 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:15.785230 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:15.785236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:15.785805 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:15.785813 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:15.786272 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:15.786275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:15.786357 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:15.786984 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:15.787004 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:15.789500 (+ 2496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":95,"threads_started":1}
W20260806 04:40:15.789743 9844 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:15.783485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:39541 (local address 127.31.166.190:40931)
0806 04:40:15.783644 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:15.783648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:15.783779 (+ 131us) server_negotiation.cc:407] Connection header received
0806 04:40:15.783971 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:15.783975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:15.784030 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:15.784116 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:15.784595 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:15.785122 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:15.785925 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:15.786158 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:15.786266 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:15.789624 (+ 3358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:39541: BlockingRecv error: recv got EOF from 127.31.166.131:39541 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 04:40:16 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:16 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:16.401410 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:16.395611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:57991)
0806 04:40:16.395905 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:16.395919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:16.396007 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:16.396339 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:16.396342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:16.396535 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:16.396636 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:16.396641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:16.397538 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:16.397540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:16.398086 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:16.398093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:16.398639 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:16.398643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:16.398705 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:16.399341 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:16.399354 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:16.401215 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":72,"threads_started":1}
W20260806 04:40:16.401643 9848 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:16.395674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:57991 (local address 127.31.166.190:40931)
0806 04:40:16.395957 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:16.395961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:16.395974 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:40:16.396140 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:16.396143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:16.396202 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:16.396317 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:16.396775 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:16.397415 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:16.398208 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:16.398492 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:16.398592 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:16.401437 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:57991: BlockingRecv error: recv got EOF from 127.31.166.129:57991 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":102,"threads_started":1}
Aug 06 04:40:16 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:16 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:16.584511 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:16.578639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:46539)
0806 04:40:16.578973 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:16.578990 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:16.579145 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:16.579443 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:16.579446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:16.579706 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:16.579825 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:16.579834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:16.580625 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:16.580630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:16.581211 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:16.581219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:16.581687 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:16.581691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:16.581789 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:16.582307 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:16.582320 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:16.584371 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
W20260806 04:40:16.584659 9848 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:16.578753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:46539 (local address 127.31.166.190:40931)
0806 04:40:16.578940 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:16.578945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:16.579101 (+ 156us) server_negotiation.cc:407] Connection header received
0806 04:40:16.579256 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:16.579260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:16.579320 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:16.579412 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:16.579941 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:16.580518 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:16.581339 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:16.581564 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:16.581671 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:16.584525 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:46539: BlockingRecv error: recv got EOF from 127.31.166.130:46539 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 06 04:40:16 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:16 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:16.796554 9850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:16.790765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:49279)
0806 04:40:16.791132 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:16.791150 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:16.791259 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:16.791535 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:16.791537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:16.791718 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:16.791817 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:16.791822 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:16.792668 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:16.792671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:16.793246 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:16.793255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:16.793758 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:16.793760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:16.793831 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:16.794335 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:16.794347 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:16.796394 (+ 2047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":103,"threads_started":1}
W20260806 04:40:16.796671 9848 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:16.790986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:49279 (local address 127.31.166.190:40931)
0806 04:40:16.791160 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:16.791165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:16.791206 (+ 41us) server_negotiation.cc:407] Connection header received
0806 04:40:16.791372 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:16.791374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:16.791422 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:16.791493 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:16.791949 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:16.792546 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:16.793420 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:16.793651 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:16.793738 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:16.796561 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:49279: BlockingRecv error: recv got EOF from 127.31.166.131:49279 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 06 04:40:17 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:17 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:17.408331 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:17.402288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:36145)
0806 04:40:17.402645 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:17.402661 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:17.402781 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:17.403167 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:17.403170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:17.403380 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:17.403503 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:17.403509 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:17.404301 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:17.404304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:17.404886 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:17.404893 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:17.405310 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:17.405312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:17.405396 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:17.405906 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:17.405918 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:17.408156 (+ 2238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":103,"threads_started":1}
W20260806 04:40:17.408486 9852 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:17.402426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:36145 (local address 127.31.166.190:40931)
0806 04:40:17.402684 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:17.402687 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:17.402719 (+ 32us) server_negotiation.cc:407] Connection header received
0806 04:40:17.402904 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:17.402907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:17.402959 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:17.403038 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:17.403648 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:17.404195 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:17.404990 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:17.405212 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:17.405289 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:17.408324 (+ 3035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:36145: BlockingRecv error: recv got EOF from 127.31.166.129:36145 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
Aug 06 04:40:17 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:17 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:17.599654 9853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:17.585352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:44801)
0806 04:40:17.585799 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:17.585812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:17.585936 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:17.586259 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:17.586262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:17.586503 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:17.586655 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:17.586666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:17.587801 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:17.587806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:17.588627 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:17.588638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:17.596116 (+ 7478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:17.596120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:17.596216 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:17.596976 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:17.596996 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:17.599475 (+ 2479us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":119,"threads_started":1}
W20260806 04:40:17.599747 9852 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:17.585517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:44801 (local address 127.31.166.190:40931)
0806 04:40:17.585662 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:17.585665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:17.585889 (+ 224us) server_negotiation.cc:407] Connection header received
0806 04:40:17.586049 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:17.586052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:17.586109 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:17.586186 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:17.586771 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:17.587669 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:17.595363 (+ 7694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:17.595755 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:17.595920 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:17.599625 (+ 3705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:44801: BlockingRecv error: recv got EOF from 127.31.166.130:44801 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 04:40:17 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:17 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:17.802735 9854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:17.797417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:55129)
0806 04:40:17.797691 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:17.797705 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:17.797799 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:17.798037 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:17.798039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:17.798220 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:17.798317 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:17.798322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:17.799059 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:17.799061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:17.799694 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:17.799703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:17.800133 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:17.800136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:17.800196 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:17.800709 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:17.800720 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:17.802594 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":105,"threads_started":1}
W20260806 04:40:17.802862 9852 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:17.797510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:55129 (local address 127.31.166.190:40931)
0806 04:40:17.797659 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:17.797663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:17.797772 (+ 109us) server_negotiation.cc:407] Connection header received
0806 04:40:17.797878 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:17.797881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:17.797926 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:17.797997 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:17.798417 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:17.798961 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:17.799818 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:17.800035 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:17.800115 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:17.802722 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:55129: BlockingRecv error: recv got EOF from 127.31.166.131:55129 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 04:40:18 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:18 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:18.415212 9855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:18.409339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:35661)
0806 04:40:18.409710 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:18.409728 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:18.409860 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:18.410164 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:18.410167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:18.410459 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:18.410558 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:18.410564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:18.411440 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:18.411445 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:18.412007 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:18.412022 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:18.412485 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:18.412489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:18.412577 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:18.413137 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:18.413158 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:18.415001 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":91,"threads_started":1}
W20260806 04:40:18.415271 9856 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:18.409403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:35661 (local address 127.31.166.190:40931)
0806 04:40:18.409680 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:18.409684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:18.409817 (+ 133us) server_negotiation.cc:407] Connection header received
0806 04:40:18.409994 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:18.409996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:18.410042 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:18.410138 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:18.410694 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:18.411332 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:18.412133 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:18.412368 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:18.412479 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:18.415148 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:35661: BlockingRecv error: recv got EOF from 127.31.166.129:35661 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":66,"threads_started":1}
Aug 06 04:40:18 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:18 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:18.606370 9857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:18.600510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:37663)
0806 04:40:18.600792 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:18.600807 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:18.600917 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:18.601178 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:18.601181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:18.601434 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:18.601537 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:18.601543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:18.602341 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:18.602344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:18.602954 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:18.602962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:18.603464 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:18.603467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:18.603538 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:18.604055 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:18.604071 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:18.606170 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":71,"threads_started":1}
W20260806 04:40:18.606433 9856 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:18.600665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:37663 (local address 127.31.166.190:40931)
0806 04:40:18.600826 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:18.600830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:18.600878 (+ 48us) server_negotiation.cc:407] Connection header received
0806 04:40:18.601007 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:18.601009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:18.601054 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:18.601121 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:18.601653 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:18.602190 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:18.603109 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:18.603339 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:18.603437 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:18.606324 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:37663: BlockingRecv error: recv got EOF from 127.31.166.130:37663 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 04:40:18 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:18 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:18.814458 9858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:18.803541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:41467)
0806 04:40:18.803866 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:18.803883 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:18.803993 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:18.804205 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:18.804207 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:18.804456 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:18.804554 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:18.804559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:18.805359 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:18.805362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:18.805934 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:18.805942 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:18.806368 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:18.806370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:18.806432 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:18.812106 (+ 5674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:18.812125 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:18.814301 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
W20260806 04:40:18.814633 9856 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:18.803665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:41467 (local address 127.31.166.190:40931)
0806 04:40:18.803853 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:18.803857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:18.803982 (+ 125us) server_negotiation.cc:407] Connection header received
0806 04:40:18.804067 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:18.804069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:18.804119 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:18.804191 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:18.804697 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:18.805246 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:18.806043 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:18.806263 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:18.806353 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:18.814488 (+ 8135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:41467: BlockingRecv error: recv got EOF from 127.31.166.131:41467 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 06 04:40:19 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:19 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:19.421490 9859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:19.416078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:37371)
0806 04:40:19.416352 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:19.416366 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:19.416459 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:19.416706 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:19.416708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:19.416900 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:19.416998 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:19.417003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:19.417754 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:19.417756 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:19.418311 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:19.418319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:19.418796 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:19.418799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:19.418878 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:19.419402 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:19.419413 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:19.421303 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
W20260806 04:40:19.421566 9860 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:19.416139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:37371 (local address 127.31.166.190:40931)
0806 04:40:19.416413 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:19.416417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:19.416433 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:40:19.416546 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:19.416548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:19.416592 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:19.416669 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:19.417100 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:19.417659 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:19.418463 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:19.418695 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:19.418775 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:19.421431 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:37371: BlockingRecv error: recv got EOF from 127.31.166.129:37371 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
Aug 06 04:40:19 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:19 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:19.613323 9861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:19.607316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:59067)
0806 04:40:19.607631 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:19.607646 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:19.607754 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:19.608014 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:19.608016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:19.608223 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:19.608371 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:19.608377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:19.609119 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:19.609121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:19.609881 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:19.609892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:19.610556 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:19.610561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:19.610647 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:19.611296 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:19.611307 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:19.613161 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":115,"threads_started":1}
W20260806 04:40:19.613413 9860 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:19.607460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:59067 (local address 127.31.166.190:40931)
0806 04:40:19.607635 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:19.607639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:19.607701 (+ 62us) server_negotiation.cc:407] Connection header received
0806 04:40:19.607840 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:19.607842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:19.607894 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:19.607964 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:19.608477 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:19.609014 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:19.610008 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:19.610355 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:19.610487 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:19.613293 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:59067: BlockingRecv error: recv got EOF from 127.31.166.130:59067 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 04:40:19 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:19 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:19.821599 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:19.815478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:40897)
0806 04:40:19.815804 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:19.815817 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:19.815970 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:19.816248 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:19.816251 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:19.816496 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:19.816588 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:19.816594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:19.817358 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:19.817373 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:19.817924 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:19.817932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:19.818392 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:19.818396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:19.818482 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:19.819054 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:19.819066 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:19.821470 (+ 2404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":137,"threads_started":1}
W20260806 04:40:19.821728 9860 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:19.815540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:40897 (local address 127.31.166.190:40931)
0806 04:40:19.815707 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:19.815712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:19.815908 (+ 196us) server_negotiation.cc:407] Connection header received
0806 04:40:19.816094 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:19.816097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:19.816145 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:19.816226 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:19.816717 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:19.817251 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:19.818046 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:19.818272 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:19.818380 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:19.821609 (+ 3229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:40897: BlockingRecv error: recv got EOF from 127.31.166.131:40897 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 04:40:20 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:20 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:20.428785 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:20.422410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:55703)
0806 04:40:20.422761 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:20.422780 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:20.422923 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:20.423271 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:20.423275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:20.423514 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:20.423617 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:20.423624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:20.424498 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:20.424503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:20.425122 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:20.425130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:20.425654 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:20.425658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:20.425755 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:20.426384 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:20.426397 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:20.428561 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":117,"threads_started":1}
W20260806 04:40:20.428913 9865 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:20.422427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:55703 (local address 127.31.166.190:40931)
0806 04:40:20.422861 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:20.422866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:20.422884 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:40:20.423036 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:20.423039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:20.423135 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:20.423241 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:20.423754 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:20.424371 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:20.425251 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:20.425513 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:20.425636 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:20.428747 (+ 3111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:55703: BlockingRecv error: recv got EOF from 127.31.166.129:55703 (error 108)
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":90,"threads_started":1}
Aug 06 04:40:20 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:20 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:20.619676 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:20.614331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:41081)
0806 04:40:20.614652 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:20.614667 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:20.614771 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:20.615009 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:20.615011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:20.615231 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:20.615329 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:20.615336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:20.616074 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:20.616076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:20.616632 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:20.616639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:20.617059 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:20.617061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:20.617120 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:20.617654 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:20.617664 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:20.619547 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":129,"threads_started":1}
W20260806 04:40:20.619823 9865 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:20.614394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:41081 (local address 127.31.166.190:40931)
0806 04:40:20.614586 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:20.614592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:20.614766 (+ 174us) server_negotiation.cc:407] Connection header received
0806 04:40:20.614839 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:20.614841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:20.614899 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:20.614975 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:20.615448 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:20.615979 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:20.616738 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:20.616963 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:20.617042 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:20.619691 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:41081: BlockingRecv error: recv got EOF from 127.31.166.130:41081 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 04:40:20 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:20 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.131@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:20.828709 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:20.822598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.131:33789)
0806 04:40:20.822970 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:20.823002 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:20.823162 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:20.823484 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:20.823487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:20.823733 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:20.823879 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:20.823888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:20.824960 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:20.824963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:20.825544 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:20.825552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:20.826015 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:20.826018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:20.826079 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:20.826640 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:20.826652 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:20.828584 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":119,"threads_started":1}
W20260806 04:40:20.828845 9865 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:20.822723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:33789 (local address 127.31.166.190:40931)
0806 04:40:20.822866 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:20.822872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:20.823124 (+ 252us) server_negotiation.cc:407] Connection header received
0806 04:40:20.823257 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:20.823262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:20.823361 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:20.823456 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:20.823994 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:20.824824 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:20.825684 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:20.825911 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:20.826000 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:20.828723 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.131:33789: BlockingRecv error: recv got EOF from 127.31.166.131:33789 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 04:40:21 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:21 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.129@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:21.435881 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:21.429855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.129:36753)
0806 04:40:21.430208 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:21.430238 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:21.430335 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:21.430616 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:21.430618 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:21.430805 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:21.430904 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:21.430909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:21.431881 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:21.431884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:21.432500 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:21.432509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:21.432969 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:21.432971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:21.433033 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:21.433582 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:21.433601 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:21.435691 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":91,"threads_started":1}
W20260806 04:40:21.435952 9869 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:21.429962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:36753 (local address 127.31.166.190:40931)
0806 04:40:21.430305 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:21.430308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:21.430365 (+ 57us) server_negotiation.cc:407] Connection header received
0806 04:40:21.430453 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:21.430456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:21.430501 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:21.430574 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:21.431043 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:21.431758 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:21.432625 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:21.432859 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:21.432944 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:21.435821 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.129:36753: BlockingRecv error: recv got EOF from 127.31.166.129:36753 (error 108)
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":107,"threads_started":1}
Aug 06 04:40:21 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 04:40:21 dist-test-slave-vkjp krb5kdc[8728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.166.130@KRBTEST.COM for oryx/127.31.166.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 04:40:21.626350 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:21.620776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40931 (local address 127.31.166.130:39893)
0806 04:40:21.621085 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:21.621099 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:21.621198 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:21.621479 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:21.621481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:21.621676 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:21.621783 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:21.621790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:21.622542 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:21.622545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:21.623219 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:21.623229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:21.623715 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:21.623718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:21.623779 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:21.624254 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:21.624266 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:21.626179 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.166.190:40931: Server oryx/127.31.166.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":131,"threads_started":1}
W20260806 04:40:21.626428 9869 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:21.620878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:39893 (local address 127.31.166.190:40931)
0806 04:40:21.621045 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:21.621052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:21.621170 (+ 118us) server_negotiation.cc:407] Connection header received
0806 04:40:21.621286 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:21.621289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:21.621333 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:21.621432 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:21.621886 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:21.622422 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:21.623361 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:21.623614 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:21.623697 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:21.626305 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.166.130:39893: BlockingRecv error: recv got EOF from 127.31.166.130:39893 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:40:21.703333 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 9365
I20260806 04:40:21.709959 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 9527
I20260806 04:40:21.717536 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 9665
I20260806 04:40:21.724046 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 9290
2026-08-06T04:40:21Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18146 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T04:40:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:40:21Z Disabled control of system clock
I20260806 04:40:21.755573 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:35031
--webserver_interface=127.31.166.190
--webserver_port=0
--builtin_ntp_servers=127.31.166.148:39827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:35031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 04:40:21.866349 9874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:21.866611 9874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:21.866645 9874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:21.870605 9874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:40:21.870658 9874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:21.870674 9874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:21.870692 9874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:40:21.870707 9874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:40:21.875340 9874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:39827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:35031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:35031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:21.876338 9874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:21.877714 9874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:21.886224 9880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:21.886384 9874 server_base.cc:1034] running on GCE node
W20260806 04:40:21.886211 9881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:21.886209 9883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:21.887061 9874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:21.888199 9874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:21.889356 9874 hybrid_clock.cc:648] HybridClock initialized: now 1785991221889330 us; error 40 us; skew 500 ppm
I20260806 04:40:21.891245 9874 webserver.cc:460] Webserver started at http://127.31.166.190:42101/ using document root <none> and password file <none>
I20260806 04:40:21.891783 9874 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:21.891842 9874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:21.892012 9874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:21.894037 9874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "edbece18aac146cea36928cd15599aec"
format_stamp: "Formatted at 2026-08-06 04:40:21 on dist-test-slave-vkjp"
server_key: "fd3b45d50c1ef9cd815f92886cd27165"
server_key_iv: "0ff07c08504dde5384750a8abbc4f0ad"
server_key_version: "encryptionkey@0"
I20260806 04:40:21.894557 9874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "edbece18aac146cea36928cd15599aec"
format_stamp: "Formatted at 2026-08-06 04:40:21 on dist-test-slave-vkjp"
server_key: "fd3b45d50c1ef9cd815f92886cd27165"
server_key_iv: "0ff07c08504dde5384750a8abbc4f0ad"
server_key_version: "encryptionkey@0"
I20260806 04:40:21.898221 9874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260806 04:40:21.900691 9889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:21.902081 9874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:40:21.902190 9874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "edbece18aac146cea36928cd15599aec"
format_stamp: "Formatted at 2026-08-06 04:40:21 on dist-test-slave-vkjp"
server_key: "fd3b45d50c1ef9cd815f92886cd27165"
server_key_iv: "0ff07c08504dde5384750a8abbc4f0ad"
server_key_version: "encryptionkey@0"
I20260806 04:40:21.902283 9874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:21.911507 9874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:21.912146 9874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:21.912300 9874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:21.919482 9874 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:35031
I20260806 04:40:21.919492 9946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:35031 every 8 connection(s)
I20260806 04:40:21.920466 9874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:40:21.921247 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 9874
I20260806 04:40:21.921384 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:40:21.921689 32410 external_mini_cluster.cc:1428] Setting key d7116fff2634d3e7ab75b8a246f85b4f
I20260806 04:40:21.924084 9947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:21.929674 9947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec: Bootstrap starting.
I20260806 04:40:21.931005 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:21.923219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.31.166.190:35031)
0806 04:40:21.923876 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:21.923890 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:21.923925 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:40:21.924619 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:21.924635 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:21.925014 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:21.925339 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:21.925754 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:21.926711 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:21.927663 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:21.928138 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:21.928310 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:21.929249 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:21.929253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:21.929265 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:40:21.929651 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:21.929701 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:40:21.929720 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:21.929848 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:21.930019 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 04:40:21.930306 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":160,"threads_started":1}
I20260806 04:40:21.932050 9947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:21.932917 9947 log.cc:824] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec: Log is configured to *not* fsync() on all Append() calls
I20260806 04:40:21.934772 9947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec: No bootstrap required, opened a new log
I20260806 04:40:21.938692 9947 raft_consensus.cc:348] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbece18aac146cea36928cd15599aec" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35031 } }
I20260806 04:40:21.938952 9947 raft_consensus.cc:374] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:21.939023 9947 raft_consensus.cc:729] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edbece18aac146cea36928cd15599aec, State: Initialized, Role: FOLLOWER
I20260806 04:40:21.939483 9947 consensus_queue.cc:260] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbece18aac146cea36928cd15599aec" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35031 } }
I20260806 04:40:21.939642 9947 raft_consensus.cc:388] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:40:21.939714 9947 raft_consensus.cc:482] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:40:21.939818 9947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:21.940860 9947 raft_consensus.cc:504] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbece18aac146cea36928cd15599aec" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35031 } }
I20260806 04:40:21.941263 9947 leader_election.cc:302] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edbece18aac146cea36928cd15599aec; no voters:
I20260806 04:40:21.941612 9947 leader_election.cc:288] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:40:21.941781 9952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:40:21.942075 9952 raft_consensus.cc:686] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [term 1 LEADER]: Becoming Leader. State: Replica: edbece18aac146cea36928cd15599aec, State: Running, Role: LEADER
I20260806 04:40:21.942453 9952 consensus_queue.cc:237] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbece18aac146cea36928cd15599aec" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35031 } }
I20260806 04:40:21.943212 9947 sys_catalog.cc:564] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:40:21.944314 9954 sys_catalog.cc:455] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [sys.catalog]: SysCatalogTable state changed. Reason: New leader edbece18aac146cea36928cd15599aec. Latest consensus state: current_term: 1 leader_uuid: "edbece18aac146cea36928cd15599aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbece18aac146cea36928cd15599aec" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35031 } } }
I20260806 04:40:21.944358 9953 sys_catalog.cc:455] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edbece18aac146cea36928cd15599aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbece18aac146cea36928cd15599aec" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35031 } } }
I20260806 04:40:21.944480 9954 sys_catalog.cc:458] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [sys.catalog]: This master's current role is: LEADER
I20260806 04:40:21.944484 9953 sys_catalog.cc:458] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec [sys.catalog]: This master's current role is: LEADER
W20260806 04:40:21.949493 9968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:40:21.949626 9968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:40:21.949760 9962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:40:21.950894 9962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:40:21.962508 9962 catalog_manager.cc:1347] Generated new cluster ID: 094fb1a527dd46cc86df927462ba13bd
I20260806 04:40:21.962808 9962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:40:21.995708 9962 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:40:21.997644 9962 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:40:22.006423 9962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edbece18aac146cea36928cd15599aec: Generated new TSK 0
I20260806 04:40:22.008297 9962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:40:22.012720 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:35031
--builtin_ntp_servers=127.31.166.148:39827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 04:40:22.189821 9972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:22.190109 9972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:22.190181 9972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:22.194319 9972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:22.194398 9972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:22.194484 9972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:40:22.200188 9972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:39827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:35031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:22.201401 9972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:22.202881 9972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:22.212706 9981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:22.212867 9972 server_base.cc:1034] running on GCE node
W20260806 04:40:22.212698 9978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:22.212715 9977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:22.213342 9972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:22.214008 9972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:22.215163 9972 hybrid_clock.cc:648] HybridClock initialized: now 1785991222215144 us; error 33 us; skew 500 ppm
I20260806 04:40:22.217087 9972 webserver.cc:460] Webserver started at http://127.31.166.129:43627/ using document root <none> and password file <none>
I20260806 04:40:22.217675 9972 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:22.217769 9972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:22.218008 9972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:22.220089 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "116de146fbeb448ea92568c4993a9405"
format_stamp: "Formatted at 2026-08-06 04:40:22 on dist-test-slave-vkjp"
server_key: "724b168864dbcc38ab59f13f5079e3b8"
server_key_iv: "edbaf9f6f5e46036a26a6e9eac271dc1"
server_key_version: "encryptionkey@0"
I20260806 04:40:22.220654 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "116de146fbeb448ea92568c4993a9405"
format_stamp: "Formatted at 2026-08-06 04:40:22 on dist-test-slave-vkjp"
server_key: "724b168864dbcc38ab59f13f5079e3b8"
server_key_iv: "edbaf9f6f5e46036a26a6e9eac271dc1"
server_key_version: "encryptionkey@0"
I20260806 04:40:22.224172 9972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:40:22.226572 9990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:22.227797 9972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:40:22.227931 9972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "116de146fbeb448ea92568c4993a9405"
format_stamp: "Formatted at 2026-08-06 04:40:22 on dist-test-slave-vkjp"
server_key: "724b168864dbcc38ab59f13f5079e3b8"
server_key_iv: "edbaf9f6f5e46036a26a6e9eac271dc1"
server_key_version: "encryptionkey@0"
I20260806 04:40:22.228044 9972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:22.247786 9972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:22.248540 9972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:22.248764 9972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:22.249364 9972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:22.250432 9972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:22.250501 9972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:22.250579 9972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:22.250622 9972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:22.259511 9972 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:35533
I20260806 04:40:22.259536 10108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:35533 every 8 connection(s)
I20260806 04:40:22.260433 9972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:40:22.263983 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 9972
I20260806 04:40:22.264106 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:40:22.264509 32410 external_mini_cluster.cc:1428] Setting key 58613ca24ef1e6128173db157a53c992
I20260806 04:40:22.266719 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:35031
--builtin_ntp_servers=127.31.166.148:39827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 04:40:22.270885 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:22.262278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:56577 (local address 127.31.166.190:35031)
0806 04:40:22.262425 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:22.262429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:22.263546 (+ 1117us) server_negotiation.cc:407] Connection header received
0806 04:40:22.265413 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:22.265416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:22.265471 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:22.265556 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:22.266695 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:22.267374 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:22.268844 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:22.269080 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:22.269191 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:22.270358 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:22.270362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:22.270365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:40:22.270378 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:22.270408 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:40:22.270415 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:22.270492 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:22.270657 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 04:40:22.270741 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:40:22.271737 10111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:22.262677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:35031 (local address 127.31.166.129:56577)
0806 04:40:22.263395 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:22.263436 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:22.265216 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:22.265719 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:22.265726 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:22.265857 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:22.266533 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:22.266557 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:22.267508 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:22.267514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:22.268698 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:22.268708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:22.269235 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:22.269239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:22.269409 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:22.270045 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:40:22.270046 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:22.270117 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:40:22.270202 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:22.270506 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:22.270516 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:22.270521 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:40:22.270656 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 04:40:22.270941 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":123,"threads_started":1}
I20260806 04:40:22.273025 10109 heartbeater.cc:347] Connected to a master server at 127.31.166.190:35031
I20260806 04:40:22.273270 10109 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:22.273772 10109 heartbeater.cc:511] Master 127.31.166.190:35031 requested a full tablet report, sending...
I20260806 04:40:22.275463 9909 ts_manager.cc:194] Registered new tserver with Master: 116de146fbeb448ea92568c4993a9405 (127.31.166.129:35533)
I20260806 04:40:22.277240 9909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.166.129:56577
W20260806 04:40:22.379853 10112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:22.380106 10112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:22.380144 10112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:22.384140 10112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:22.384189 10112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:22.384240 10112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:40:22.388770 10112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:39827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:35031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:22.389783 10112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:22.391135 10112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:22.400569 10118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:22.400645 10117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:22.400665 10112 server_base.cc:1034] running on GCE node
W20260806 04:40:22.400573 10120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:22.401429 10112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:22.402194 10112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:22.403400 10112 hybrid_clock.cc:648] HybridClock initialized: now 1785991222403370 us; error 43 us; skew 500 ppm
I20260806 04:40:22.405664 10112 webserver.cc:460] Webserver started at http://127.31.166.130:33351/ using document root <none> and password file <none>
I20260806 04:40:22.406473 10112 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:22.406596 10112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:22.406864 10112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:22.409201 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "6bbd1662ce114e0aa3fa421ab6a54cc9"
format_stamp: "Formatted at 2026-08-06 04:40:22 on dist-test-slave-vkjp"
server_key: "d1201e915d9198dc4ae120d3fa1ba5c3"
server_key_iv: "f4bf49089afd57b061f94f81e5ff4606"
server_key_version: "encryptionkey@0"
I20260806 04:40:22.409857 10112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "6bbd1662ce114e0aa3fa421ab6a54cc9"
format_stamp: "Formatted at 2026-08-06 04:40:22 on dist-test-slave-vkjp"
server_key: "d1201e915d9198dc4ae120d3fa1ba5c3"
server_key_iv: "f4bf49089afd57b061f94f81e5ff4606"
server_key_version: "encryptionkey@0"
I20260806 04:40:22.413946 10112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:40:22.416879 10126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:22.418120 10112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:40:22.418258 10112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "6bbd1662ce114e0aa3fa421ab6a54cc9"
format_stamp: "Formatted at 2026-08-06 04:40:22 on dist-test-slave-vkjp"
server_key: "d1201e915d9198dc4ae120d3fa1ba5c3"
server_key_iv: "f4bf49089afd57b061f94f81e5ff4606"
server_key_version: "encryptionkey@0"
I20260806 04:40:22.418377 10112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:22.440349 10112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:22.441097 10112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:22.441303 10112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:22.441901 10112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:22.442932 10112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:22.443001 10112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:22.443070 10112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:22.443146 10112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 04:40:22.452333 10112 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:37187
I20260806 04:40:22.452369 10246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:37187 every 8 connection(s)
I20260806 04:40:22.453236 10112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:40:22.462312 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:22.454995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:43453 (local address 127.31.166.190:35031)
0806 04:40:22.455172 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:22.455177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:22.456299 (+ 1122us) server_negotiation.cc:407] Connection header received
0806 04:40:22.457671 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:22.457674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:22.457722 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:22.457802 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:22.458654 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:22.459260 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:22.460457 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:22.460672 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:22.460758 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:22.461832 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:22.461835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:22.461837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:40:22.461849 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:22.461867 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:40:22.461874 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:22.461939 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:22.462142 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 04:40:22.462192 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 04:40:22.463266 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:22.455461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:35031 (local address 127.31.166.130:43453)
0806 04:40:22.456135 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:22.456183 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:22.457458 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:22.457955 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:22.457961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:22.458043 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:22.458513 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:22.458525 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:22.459432 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:22.459435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:22.460332 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:22.460342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:22.460823 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:22.460827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:22.460960 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:22.461563 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:40:22.461565 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:22.461600 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:40:22.461664 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:22.461979 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:22.461987 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:22.461992 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:40:22.462105 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 04:40:22.462325 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":118,"threads_started":1}
I20260806 04:40:22.463478 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 10112
I20260806 04:40:22.463589 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:40:22.463927 32410 external_mini_cluster.cc:1428] Setting key fb0a34bb77bbb2f660cb0af9d0318fe9
I20260806 04:40:22.464520 10247 heartbeater.cc:347] Connected to a master server at 127.31.166.190:35031
I20260806 04:40:22.464757 10247 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:22.465332 10247 heartbeater.cc:511] Master 127.31.166.190:35031 requested a full tablet report, sending...
I20260806 04:40:22.466109 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:35031
--builtin_ntp_servers=127.31.166.148:39827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 04:40:22.466454 9909 ts_manager.cc:194] Registered new tserver with Master: 6bbd1662ce114e0aa3fa421ab6a54cc9 (127.31.166.130:37187)
I20260806 04:40:22.467067 9909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.166.130:43453
W20260806 04:40:22.580315 10250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:22.580610 10250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:22.580685 10250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:22.584776 10250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:22.584870 10250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:22.584968 10250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:40:22.589908 10250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:39827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:35031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:22.591113 10250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:22.592543 10250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:22.602617 10256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:22.602620 10258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:22.602618 10255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:22.602880 10250 server_base.cc:1034] running on GCE node
I20260806 04:40:22.603356 10250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:22.604019 10250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:22.605208 10250 hybrid_clock.cc:648] HybridClock initialized: now 1785991222605181 us; error 47 us; skew 500 ppm
I20260806 04:40:22.607136 10250 webserver.cc:460] Webserver started at http://127.31.166.131:34629/ using document root <none> and password file <none>
I20260806 04:40:22.607738 10250 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:22.607832 10250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:22.608069 10250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:22.610276 10250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "f1d892db5cdb4cdca810c04b927af867"
format_stamp: "Formatted at 2026-08-06 04:40:22 on dist-test-slave-vkjp"
server_key: "7916f02c1b89212f3709f1e70de95fe6"
server_key_iv: "d98ae0c5935f707027c0ffbff8de1794"
server_key_version: "encryptionkey@0"
I20260806 04:40:22.610867 10250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "f1d892db5cdb4cdca810c04b927af867"
format_stamp: "Formatted at 2026-08-06 04:40:22 on dist-test-slave-vkjp"
server_key: "7916f02c1b89212f3709f1e70de95fe6"
server_key_iv: "d98ae0c5935f707027c0ffbff8de1794"
server_key_version: "encryptionkey@0"
I20260806 04:40:22.614675 10250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:40:22.617470 10264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:22.618832 10250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 04:40:22.618993 10250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "f1d892db5cdb4cdca810c04b927af867"
format_stamp: "Formatted at 2026-08-06 04:40:22 on dist-test-slave-vkjp"
server_key: "7916f02c1b89212f3709f1e70de95fe6"
server_key_iv: "d98ae0c5935f707027c0ffbff8de1794"
server_key_version: "encryptionkey@0"
I20260806 04:40:22.619163 10250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:22.638832 10250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:22.639683 10250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:22.639907 10250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:22.640568 10250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:22.641853 10250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:22.641938 10250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:22.642014 10250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:22.642072 10250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:22.652216 10250 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:44629
I20260806 04:40:22.652242 10387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:44629 every 8 connection(s)
I20260806 04:40:22.653244 10250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:40:22.663415 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 10250
I20260806 04:40:22.663445 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:22.655341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:50311 (local address 127.31.166.190:35031)
0806 04:40:22.655495 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:22.655499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:22.656704 (+ 1205us) server_negotiation.cc:407] Connection header received
0806 04:40:22.658560 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:22.658564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:22.658640 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:22.658748 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:22.659737 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:22.660284 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:22.661440 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:22.661662 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:22.661772 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:22.662897 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:22.662901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:22.662903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:40:22.662915 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:22.662933 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:40:22.662940 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:22.663014 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:22.663253 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 04:40:22.663313 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:40:22.663551 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:40:22.663898 32410 external_mini_cluster.cc:1428] Setting key 533cda0631a30b051d23dbcd27c375cc
I20260806 04:40:22.664584 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:22.655771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:35031 (local address 127.31.166.131:50311)
0806 04:40:22.656545 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:22.656603 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:22.658344 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:22.658930 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:22.658937 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:22.659025 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:22.659595 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:22.659607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:22.660413 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:22.660416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:22.661309 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:22.661318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:22.661821 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:22.661825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:22.661985 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:22.662616 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:40:22.662617 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:22.662657 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:40:22.662717 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:22.663036 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:22.663051 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:22.663055 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:40:22.663220 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 04:40:22.663539 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":135,"threads_started":1}
I20260806 04:40:22.665786 10388 heartbeater.cc:347] Connected to a master server at 127.31.166.190:35031
I20260806 04:40:22.666021 10388 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:22.666577 10388 heartbeater.cc:511] Master 127.31.166.190:35031 requested a full tablet report, sending...
I20260806 04:40:22.667537 9909 ts_manager.cc:194] Registered new tserver with Master: f1d892db5cdb4cdca810c04b927af867 (127.31.166.131:44629)
I20260806 04:40:22.668053 9909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.166.131:50311
I20260806 04:40:22.678315 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 04:40:22.681484 9950 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:22.680436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.31.166.190:35031)
0806 04:40:22.680580 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:22.680585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:22.680745 (+ 160us) server_negotiation.cc:407] Connection header received
0806 04:40:22.680926 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:22.680929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:22.680972 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:22.681073 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:22.681351 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51276: BlockingRecv error: recv got EOF from 127.0.0.1:51276 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:40:22.681978 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 9972
I20260806 04:40:22.692188 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 10112
I20260806 04:40:22.700248 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 10250
I20260806 04:40:22.707549 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 9874
2026-08-06T04:40:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (983 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T04:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:40:22Z Disabled control of system clock
I20260806 04:40:22.738926 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:38115
--webserver_interface=127.31.166.190
--webserver_port=0
--builtin_ntp_servers=127.31.166.148:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:38115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 04:40:22.853049 10399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:22.853339 10399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:22.853379 10399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:22.858172 10399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:40:22.858238 10399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:22.858254 10399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:22.858271 10399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:40:22.858286 10399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:40:22.863296 10399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:38115
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:38115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:22.864502 10399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:22.865945 10399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:22.874603 10404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:22.874605 10407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:22.874605 10405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:22.874882 10399 server_base.cc:1034] running on GCE node
I20260806 04:40:22.875556 10399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:22.876652 10399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:22.877867 10399 hybrid_clock.cc:648] HybridClock initialized: now 1785991222877847 us; error 31 us; skew 500 ppm
I20260806 04:40:22.880048 10399 webserver.cc:460] Webserver started at http://127.31.166.190:41361/ using document root <none> and password file <none>
I20260806 04:40:22.880648 10399 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:22.880749 10399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:22.881002 10399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:22.883212 10399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "79511bc85207494c8f27a9cc4f929946"
format_stamp: "Formatted at 2026-08-06 04:40:22 on dist-test-slave-vkjp"
server_key: "40e07d0586bae020408f31dd0fff687a"
server_key_iv: "d9d360b90c19d744c32992b50ba7bffe"
server_key_version: "encryptionkey@0"
I20260806 04:40:22.883797 10399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "79511bc85207494c8f27a9cc4f929946"
format_stamp: "Formatted at 2026-08-06 04:40:22 on dist-test-slave-vkjp"
server_key: "40e07d0586bae020408f31dd0fff687a"
server_key_iv: "d9d360b90c19d744c32992b50ba7bffe"
server_key_version: "encryptionkey@0"
I20260806 04:40:22.887591 10399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:40:22.890050 10413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:22.891412 10399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:40:22.891562 10399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "79511bc85207494c8f27a9cc4f929946"
format_stamp: "Formatted at 2026-08-06 04:40:22 on dist-test-slave-vkjp"
server_key: "40e07d0586bae020408f31dd0fff687a"
server_key_iv: "d9d360b90c19d744c32992b50ba7bffe"
server_key_version: "encryptionkey@0"
I20260806 04:40:22.891686 10399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:22.896659 10399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:22.897315 10399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:22.897521 10399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:22.905232 10465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:38115 every 8 connection(s)
I20260806 04:40:22.905229 10399 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:38115
I20260806 04:40:22.906383 10399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:40:22.910132 10466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:22.915457 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 10399
I20260806 04:40:22.915581 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:40:22.915921 32410 external_mini_cluster.cc:1428] Setting key 6aca572fac90ca0a6aa51bf725d54250
I20260806 04:40:22.916493 10466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946: Bootstrap starting.
I20260806 04:40:22.919170 10466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:22.919934 10466 log.cc:824] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946: Log is configured to *not* fsync() on all Append() calls
I20260806 04:40:22.922112 10466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946: No bootstrap required, opened a new log
I20260806 04:40:22.922277 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:22.917543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.31.166.190:38115)
0806 04:40:22.918329 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:22.918348 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:22.918388 (+ 40us) server_negotiation.cc:407] Connection header received
0806 04:40:22.919159 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:22.919180 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:22.919634 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:22.919991 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:22.920891 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:22.920895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:22.920907 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:40:22.920919 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:22.920978 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:40:22.921010 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:22.921139 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:22.921282 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 04:40:22.921533 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":169,"threads_started":1}
I20260806 04:40:22.926211 10466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79511bc85207494c8f27a9cc4f929946" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 38115 } }
I20260806 04:40:22.926491 10466 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:22.926579 10466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79511bc85207494c8f27a9cc4f929946, State: Initialized, Role: FOLLOWER
I20260806 04:40:22.927075 10466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79511bc85207494c8f27a9cc4f929946" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 38115 } }
I20260806 04:40:22.927299 10466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:40:22.927374 10466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:40:22.927526 10466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:22.928728 10466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79511bc85207494c8f27a9cc4f929946" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 38115 } }
I20260806 04:40:22.929107 10466 leader_election.cc:302] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79511bc85207494c8f27a9cc4f929946; no voters:
I20260806 04:40:22.929457 10466 leader_election.cc:288] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:40:22.929596 10471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:40:22.929888 10471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [term 1 LEADER]: Becoming Leader. State: Replica: 79511bc85207494c8f27a9cc4f929946, State: Running, Role: LEADER
I20260806 04:40:22.930254 10471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79511bc85207494c8f27a9cc4f929946" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 38115 } }
I20260806 04:40:22.930850 10466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:40:22.932345 10472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79511bc85207494c8f27a9cc4f929946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79511bc85207494c8f27a9cc4f929946" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 38115 } } }
I20260806 04:40:22.932345 10473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79511bc85207494c8f27a9cc4f929946. Latest consensus state: current_term: 1 leader_uuid: "79511bc85207494c8f27a9cc4f929946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79511bc85207494c8f27a9cc4f929946" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 38115 } } }
I20260806 04:40:22.932588 10473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [sys.catalog]: This master's current role is: LEADER
I20260806 04:40:22.932540 10472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946 [sys.catalog]: This master's current role is: LEADER
I20260806 04:40:22.933020 10480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:40:22.936467 10480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:40:22.943035 10480 catalog_manager.cc:1347] Generated new cluster ID: 36eb353b7758496db503a01bbb611c40
I20260806 04:40:22.943184 10480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:40:22.955050 10480 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:40:22.955556 10480 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:40:22.964519 10480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79511bc85207494c8f27a9cc4f929946: Generated new TSK 0
I20260806 04:40:22.965384 10480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:40:22.970999 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:38115
--builtin_ntp_servers=127.31.166.148:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 04:40:23.088287 10490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:23.088573 10490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:23.088614 10490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:23.092692 10490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:23.092747 10490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:23.092826 10490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:40:23.097970 10490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:38115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:23.099040 10490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:23.100482 10490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:23.110357 10498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:23.110373 10496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:23.110385 10495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:23.110659 10490 server_base.cc:1034] running on GCE node
I20260806 04:40:23.111054 10490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:23.111780 10490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:23.112982 10490 hybrid_clock.cc:648] HybridClock initialized: now 1785991223112967 us; error 28 us; skew 500 ppm
I20260806 04:40:23.114897 10490 webserver.cc:460] Webserver started at http://127.31.166.129:36119/ using document root <none> and password file <none>
I20260806 04:40:23.115532 10490 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:23.115625 10490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:23.115861 10490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:23.117939 10490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "118314db534d465496039a44108cba8d"
format_stamp: "Formatted at 2026-08-06 04:40:23 on dist-test-slave-vkjp"
server_key: "80f6444f2db314e36ef2c4b7caf5f1cd"
server_key_iv: "df11a47492e49054f7e3d3527bc97acf"
server_key_version: "encryptionkey@0"
I20260806 04:40:23.118503 10490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "118314db534d465496039a44108cba8d"
format_stamp: "Formatted at 2026-08-06 04:40:23 on dist-test-slave-vkjp"
server_key: "80f6444f2db314e36ef2c4b7caf5f1cd"
server_key_iv: "df11a47492e49054f7e3d3527bc97acf"
server_key_version: "encryptionkey@0"
I20260806 04:40:23.122232 10490 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.005s
I20260806 04:40:23.125007 10505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:23.126230 10490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:40:23.126374 10490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "118314db534d465496039a44108cba8d"
format_stamp: "Formatted at 2026-08-06 04:40:23 on dist-test-slave-vkjp"
server_key: "80f6444f2db314e36ef2c4b7caf5f1cd"
server_key_iv: "df11a47492e49054f7e3d3527bc97acf"
server_key_version: "encryptionkey@0"
I20260806 04:40:23.126493 10490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:23.131192 10490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:23.131918 10490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:23.132112 10490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:23.132712 10490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:23.133677 10490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:23.133747 10490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:23.133819 10490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:23.133867 10490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:23.142978 10490 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:46813
I20260806 04:40:23.143007 10630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:46813 every 8 connection(s)
I20260806 04:40:23.143932 10490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:40:23.148454 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 10490
I20260806 04:40:23.148619 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:40:23.149008 32410 external_mini_cluster.cc:1428] Setting key aadc6e6507993ec944d8ee9de0dfdbe7
I20260806 04:40:23.150764 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:23.145870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:41073 (local address 127.31.166.190:38115)
0806 04:40:23.146053 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:23.146057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:23.147227 (+ 1170us) server_negotiation.cc:407] Connection header received
0806 04:40:23.148659 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:23.148663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:23.148714 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:23.148799 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:23.150214 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:23.150218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:23.150220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:40:23.150234 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:23.150259 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:40:23.150269 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:23.150381 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:23.150568 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 04:40:23.150617 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 04:40:23.151402 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:38115
--builtin_ntp_servers=127.31.166.148:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 04:40:23.151966 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:23.146285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:38115 (local address 127.31.166.129:41073)
0806 04:40:23.147002 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:23.147044 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:23.148369 (+ 1325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:23.149048 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:23.149078 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:23.149201 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:23.149928 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:40:23.149930 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:23.149979 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:40:23.150051 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:23.150384 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:23.150395 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:23.150400 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 04:40:23.150575 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 04:40:23.150845 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":128,"threads_started":1}
I20260806 04:40:23.153255 10631 heartbeater.cc:347] Connected to a master server at 127.31.166.190:38115
I20260806 04:40:23.153520 10631 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:23.154031 10631 heartbeater.cc:511] Master 127.31.166.190:38115 requested a full tablet report, sending...
I20260806 04:40:23.155655 10430 ts_manager.cc:194] Registered new tserver with Master: 118314db534d465496039a44108cba8d (127.31.166.129:46813)
W20260806 04:40:23.266271 10634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:23.266588 10634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:23.266664 10634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:23.270782 10634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:23.270875 10634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:23.270977 10634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:40:23.276000 10634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:38115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:23.277148 10634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:23.278692 10634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:23.288659 10639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:23.288663 10642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:23.288657 10640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:23.288998 10634 server_base.cc:1034] running on GCE node
I20260806 04:40:23.289489 10634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:23.290134 10634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:23.291296 10634 hybrid_clock.cc:648] HybridClock initialized: now 1785991223291276 us; error 30 us; skew 500 ppm
I20260806 04:40:23.293105 10634 webserver.cc:460] Webserver started at http://127.31.166.130:46617/ using document root <none> and password file <none>
I20260806 04:40:23.293639 10634 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:23.293699 10634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:23.293866 10634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:23.295985 10634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "93ed3ba4d6d04aae91547d921c3895d0"
format_stamp: "Formatted at 2026-08-06 04:40:23 on dist-test-slave-vkjp"
server_key: "753f28f6b64bb3caf718a8a355b8fb62"
server_key_iv: "f7d3d4e9ded10de67b42f21308fa7433"
server_key_version: "encryptionkey@0"
I20260806 04:40:23.296514 10634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "93ed3ba4d6d04aae91547d921c3895d0"
format_stamp: "Formatted at 2026-08-06 04:40:23 on dist-test-slave-vkjp"
server_key: "753f28f6b64bb3caf718a8a355b8fb62"
server_key_iv: "f7d3d4e9ded10de67b42f21308fa7433"
server_key_version: "encryptionkey@0"
I20260806 04:40:23.300323 10634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:40:23.303051 10650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:23.304551 10634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:40:23.304690 10634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "93ed3ba4d6d04aae91547d921c3895d0"
format_stamp: "Formatted at 2026-08-06 04:40:23 on dist-test-slave-vkjp"
server_key: "753f28f6b64bb3caf718a8a355b8fb62"
server_key_iv: "f7d3d4e9ded10de67b42f21308fa7433"
server_key_version: "encryptionkey@0"
I20260806 04:40:23.304812 10634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:23.309494 10634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:23.310101 10634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:23.310290 10634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:23.310837 10634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:23.311821 10634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:23.311897 10634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:23.311991 10634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:23.312041 10634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:23.321111 10634 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:34227
I20260806 04:40:23.321147 10768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:34227 every 8 connection(s)
I20260806 04:40:23.322042 10634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:40:23.328208 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:23.324009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:52767 (local address 127.31.166.190:38115)
0806 04:40:23.324155 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:23.324158 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:23.325153 (+ 995us) server_negotiation.cc:407] Connection header received
0806 04:40:23.326414 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:23.326417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:23.326457 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:23.326532 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:23.327786 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:23.327789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:23.327791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:40:23.327804 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:23.327824 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:40:23.327832 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:23.327924 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:23.328049 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 04:40:23.328089 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:40:23.328685 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 10634
I20260806 04:40:23.328835 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:40:23.329233 32410 external_mini_cluster.cc:1428] Setting key 5f1502dc9c6199e0dd3282897f92d148
I20260806 04:40:23.329555 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:23.324331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:38115 (local address 127.31.166.130:52767)
0806 04:40:23.325000 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:23.325043 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:23.326220 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:23.326697 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:23.326704 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:23.326790 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:23.327470 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:40:23.327472 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:23.327512 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:40:23.327579 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:23.327919 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:23.327926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:23.327930 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:40:23.328064 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 04:40:23.328252 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":107,"threads_started":1}
I20260806 04:40:23.330891 10769 heartbeater.cc:347] Connected to a master server at 127.31.166.190:38115
I20260806 04:40:23.331197 10769 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:23.331576 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:38115
--builtin_ntp_servers=127.31.166.148:46411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 04:40:23.331652 10769 heartbeater.cc:511] Master 127.31.166.190:38115 requested a full tablet report, sending...
I20260806 04:40:23.332828 10430 ts_manager.cc:194] Registered new tserver with Master: 93ed3ba4d6d04aae91547d921c3895d0 (127.31.166.130:34227)
W20260806 04:40:23.443944 10773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:23.444187 10773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:23.444255 10773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:23.448292 10773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:23.448345 10773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:23.448436 10773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:40:23.453197 10773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:46411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:38115
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:23.454245 10773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:23.455653 10773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:23.465442 10781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:23.465426 10779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:23.465451 10778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:23.465659 10773 server_base.cc:1034] running on GCE node
I20260806 04:40:23.466200 10773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:23.466849 10773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:23.468012 10773 hybrid_clock.cc:648] HybridClock initialized: now 1785991223467980 us; error 45 us; skew 500 ppm
I20260806 04:40:23.469933 10773 webserver.cc:460] Webserver started at http://127.31.166.131:46289/ using document root <none> and password file <none>
I20260806 04:40:23.470502 10773 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:23.470566 10773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:23.470773 10773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:23.472846 10773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "03f75967e214467589ac5e7c90aaba7d"
format_stamp: "Formatted at 2026-08-06 04:40:23 on dist-test-slave-vkjp"
server_key: "305efd62987c328928290aa7a568cc8d"
server_key_iv: "2ded6872b77e1608893497ac42692ddc"
server_key_version: "encryptionkey@0"
I20260806 04:40:23.473417 10773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "03f75967e214467589ac5e7c90aaba7d"
format_stamp: "Formatted at 2026-08-06 04:40:23 on dist-test-slave-vkjp"
server_key: "305efd62987c328928290aa7a568cc8d"
server_key_iv: "2ded6872b77e1608893497ac42692ddc"
server_key_version: "encryptionkey@0"
I20260806 04:40:23.477187 10773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:40:23.479791 10787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:23.481004 10773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:40:23.481155 10773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "03f75967e214467589ac5e7c90aaba7d"
format_stamp: "Formatted at 2026-08-06 04:40:23 on dist-test-slave-vkjp"
server_key: "305efd62987c328928290aa7a568cc8d"
server_key_iv: "2ded6872b77e1608893497ac42692ddc"
server_key_version: "encryptionkey@0"
I20260806 04:40:23.481276 10773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:23.485948 10773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:23.486578 10773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:23.486756 10773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:23.487382 10773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:23.488272 10773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:23.488343 10773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:23.488411 10773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:23.488464 10773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:23.497404 10773 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:42111
I20260806 04:40:23.497474 10915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:42111 every 8 connection(s)
I20260806 04:40:23.498328 10773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:40:23.498829 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 10773
I20260806 04:40:23.498939 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:40:23.499292 32410 external_mini_cluster.cc:1428] Setting key 1a74d748b25618a30203208d8f42e6a7
I20260806 04:40:23.504711 10469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:23.500344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:55991 (local address 127.31.166.190:38115)
0806 04:40:23.500538 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:23.500543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:23.501827 (+ 1284us) server_negotiation.cc:407] Connection header received
0806 04:40:23.503043 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:23.503045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:23.503127 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:23.503213 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:23.504311 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:23.504314 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:23.504316 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:40:23.504329 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:23.504354 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:40:23.504391 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:23.504469 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:23.504575 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 04:40:23.504612 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260806 04:40:23.505550 10919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:23.500931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:38115 (local address 127.31.166.131:55991)
0806 04:40:23.501690 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:23.501731 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:23.502860 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:23.503337 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:23.503343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:23.503432 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:23.504078 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:40:23.504080 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:23.504119 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:40:23.504182 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:23.504475 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:23.504481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:23.504485 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:40:23.504609 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 04:40:23.504784 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":113,"threads_started":1}
I20260806 04:40:23.506632 10917 heartbeater.cc:347] Connected to a master server at 127.31.166.190:38115
I20260806 04:40:23.506852 10917 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:23.507254 10917 heartbeater.cc:511] Master 127.31.166.190:38115 requested a full tablet report, sending...
I20260806 04:40:23.508222 10430 ts_manager.cc:194] Registered new tserver with Master: 03f75967e214467589ac5e7c90aaba7d (127.31.166.131:42111)
I20260806 04:40:23.513348 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 04:40:23.516368 10469 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:40:23.515348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.31.166.190:38115)
0806 04:40:23.515480 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:23.515485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:23.515735 (+ 250us) server_negotiation.cc:407] Connection header received
0806 04:40:23.515845 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:23.515847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:23.515897 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:23.515965 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:23.516252 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37052: BlockingRecv error: recv got EOF from 127.0.0.1:37052 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260806 04:40:23.516561 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 10490
I20260806 04:40:23.524443 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 10634
I20260806 04:40:23.530697 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 10773
I20260806 04:40:23.536435 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 10399
2026-08-06T04:40:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (828 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:40:23 dist-test-slave-vkjp krb5kdc[10927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:40:23 dist-test-slave-vkjp krb5kdc[10927](info): set up 2 sockets
Aug 06 04:40:23 dist-test-slave-vkjp krb5kdc[10927](info): commencing operation
krb5kdc: starting...
W20260806 04:40:24.687561 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:40:24 dist-test-slave-vkjp krb5kdc[10927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:40:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:40:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:40:24.824018 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:44437
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:44437
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:40:24.936460 10941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:24.936764 10941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:24.936841 10941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:24.941063 10941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:40:24.941160 10941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:24.941193 10941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:24.941232 10941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:40:24.941267 10941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:40:24.947367 10941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:44437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:44437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:24.948529 10941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:24.949653 10941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:24.958256 10949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:24.958267 10946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:24.958315 10947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:24.958580 10941 server_base.cc:1034] running on GCE node
I20260806 04:40:24.959267 10941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:24.960466 10941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:24.961694 10941 hybrid_clock.cc:648] HybridClock initialized: now 1785991224961650 us; error 52 us; skew 500 ppm
Aug 06 04:40:24 dist-test-slave-vkjp krb5kdc[10927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991224, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:24.968504 10941 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:40:24.969794 10941 webserver.cc:460] Webserver started at http://127.31.166.190:35511/ using document root <none> and password file <none>
I20260806 04:40:24.970458 10941 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:24.970523 10941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:24.970757 10941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:24.973002 10941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "5e86166fa3a2401a8604513271cbab5b"
format_stamp: "Formatted at 2026-08-06 04:40:24 on dist-test-slave-vkjp"
server_key: "837217029f8e3abec970b6ff65951840"
server_key_iv: "6372f429bbf95b28e0352deb0c9138a1"
server_key_version: "encryptionkey@0"
I20260806 04:40:24.973610 10941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "5e86166fa3a2401a8604513271cbab5b"
format_stamp: "Formatted at 2026-08-06 04:40:24 on dist-test-slave-vkjp"
server_key: "837217029f8e3abec970b6ff65951840"
server_key_iv: "6372f429bbf95b28e0352deb0c9138a1"
server_key_version: "encryptionkey@0"
I20260806 04:40:24.977610 10941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:40:24.980484 10957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:24.981895 10941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:40:24.982048 10941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "5e86166fa3a2401a8604513271cbab5b"
format_stamp: "Formatted at 2026-08-06 04:40:24 on dist-test-slave-vkjp"
server_key: "837217029f8e3abec970b6ff65951840"
server_key_iv: "6372f429bbf95b28e0352deb0c9138a1"
server_key_version: "encryptionkey@0"
I20260806 04:40:24.982170 10941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:24.998128 10941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:25.001652 10941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:25.001880 10941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:25.009867 10941 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:44437
I20260806 04:40:25.009930 11015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:44437 every 8 connection(s)
I20260806 04:40:25.010998 10941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:40:25.014022 11016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:25.019915 11016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b: Bootstrap starting.
I20260806 04:40:25.021229 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 10941
I20260806 04:40:25.021335 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:40:25.021637 32410 external_mini_cluster.cc:1428] Setting key a9583d28b5a41094e35a9cd54fbf326a
I20260806 04:40:25.022733 11016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:25.023649 11016 log.cc:824] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b: Log is configured to *not* fsync() on all Append() calls
I20260806 04:40:25.025893 11016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b: No bootstrap required, opened a new log
I20260806 04:40:25.029698 11016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e86166fa3a2401a8604513271cbab5b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44437 } }
I20260806 04:40:25.029959 11016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:25.030025 11016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e86166fa3a2401a8604513271cbab5b, State: Initialized, Role: FOLLOWER
Aug 06 04:40:25 dist-test-slave-vkjp krb5kdc[10927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:40:25.030525 11016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e86166fa3a2401a8604513271cbab5b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44437 } }
I20260806 04:40:25.030730 11016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:40:25.030802 11016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:40:25.030906 11016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:25.032316 11016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e86166fa3a2401a8604513271cbab5b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44437 } }
I20260806 04:40:25.032744 11016 leader_election.cc:302] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e86166fa3a2401a8604513271cbab5b; no voters:
I20260806 04:40:25.033141 11016 leader_election.cc:288] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:40:25.033270 11022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:40:25.033519 11022 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [term 1 LEADER]: Becoming Leader. State: Replica: 5e86166fa3a2401a8604513271cbab5b, State: Running, Role: LEADER
I20260806 04:40:25.033906 11022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e86166fa3a2401a8604513271cbab5b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44437 } }
I20260806 04:40:25.034629 11016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:40:25.035722 11023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e86166fa3a2401a8604513271cbab5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e86166fa3a2401a8604513271cbab5b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44437 } } }
I20260806 04:40:25.035857 11023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [sys.catalog]: This master's current role is: LEADER
I20260806 04:40:25.036173 11024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e86166fa3a2401a8604513271cbab5b. Latest consensus state: current_term: 1 leader_uuid: "5e86166fa3a2401a8604513271cbab5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e86166fa3a2401a8604513271cbab5b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44437 } } }
I20260806 04:40:25.036237 11024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b [sys.catalog]: This master's current role is: LEADER
I20260806 04:40:25.037137 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:25.022857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.31.166.190:44437)
0806 04:40:25.023327 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:25.023342 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:25.023384 (+ 42us) server_negotiation.cc:407] Connection header received
0806 04:40:25.024146 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:25.024170 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:25.024582 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:25.024916 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:25.025752 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.026797 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.027740 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.028255 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.028442 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:25.031291 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:25.031298 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:25.031314 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:25.031348 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:25.033494 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:25.034061 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:25.034067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:25.034072 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:25.034209 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:25.034552 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:25.034556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:25.034558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:25.035147 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:25.035359 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:25.035675 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 04:40:25.036018 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":178,"threads_started":1}
W20260806 04:40:25.039686 11038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:40:25.039770 11038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:40:25.039848 11031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:40:25.040577 11031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:40:25.046314 11031 catalog_manager.cc:1347] Generated new cluster ID: b961ffcd89804ac4846254b61e836338
I20260806 04:40:25.046393 11031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:40:25.057461 11031 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:40:25.058444 11031 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:40:25.068953 11031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b: Generated new TSK 0
I20260806 04:40:25.069669 11031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:40:25.102558 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:44437
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:40:25.254249 11044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:25.254516 11044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:25.254559 11044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:25.258741 11044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:25.258806 11044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:25.258924 11044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:40:25.264250 11044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:44437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:25.265344 11044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:25.266441 11044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:25.276274 11052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:25.276266 11049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:25.276269 11050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:25.276311 11044 server_base.cc:1034] running on GCE node
I20260806 04:40:25.277004 11044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:25.277772 11044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:25.278995 11044 hybrid_clock.cc:648] HybridClock initialized: now 1785991225278960 us; error 46 us; skew 500 ppm
Aug 06 04:40:25 dist-test-slave-vkjp krb5kdc[10927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991225, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:25.281901 11044 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:40:25.282801 11044 webserver.cc:460] Webserver started at http://127.31.166.129:40513/ using document root <none> and password file <none>
I20260806 04:40:25.283456 11044 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:25.283550 11044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:25.283785 11044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:25.285967 11044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "849a45448aaa4fbda6350eefaf14a41d"
format_stamp: "Formatted at 2026-08-06 04:40:25 on dist-test-slave-vkjp"
server_key: "2fd3fbfe881b1b04a0abce4d61e6e1a6"
server_key_iv: "ef1b1c072c9801043e20b57e3bdda026"
server_key_version: "encryptionkey@0"
I20260806 04:40:25.286614 11044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "849a45448aaa4fbda6350eefaf14a41d"
format_stamp: "Formatted at 2026-08-06 04:40:25 on dist-test-slave-vkjp"
server_key: "2fd3fbfe881b1b04a0abce4d61e6e1a6"
server_key_iv: "ef1b1c072c9801043e20b57e3bdda026"
server_key_version: "encryptionkey@0"
I20260806 04:40:25.290341 11044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 04:40:25.292845 11059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:25.294081 11044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:40:25.294216 11044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "849a45448aaa4fbda6350eefaf14a41d"
format_stamp: "Formatted at 2026-08-06 04:40:25 on dist-test-slave-vkjp"
server_key: "2fd3fbfe881b1b04a0abce4d61e6e1a6"
server_key_iv: "ef1b1c072c9801043e20b57e3bdda026"
server_key_version: "encryptionkey@0"
I20260806 04:40:25.294327 11044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:25.305893 11044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:25.309549 11044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:25.309751 11044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:25.310354 11044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:25.311192 11044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:25.311241 11044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:25.311282 11044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:25.311298 11044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:25.319900 11044 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:40901
I20260806 04:40:25.319981 11181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:40901 every 8 connection(s)
I20260806 04:40:25.320935 11044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:40:25.330427 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 11044
I20260806 04:40:25.330533 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
Aug 06 04:40:25 dist-test-slave-vkjp krb5kdc[10927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991225, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:40:25.330884 32410 external_mini_cluster.cc:1428] Setting key 05f9d1d4a231312e8a81e4674bcccb8c
I20260806 04:40:25.335999 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:25.322826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:53461 (local address 127.31.166.190:44437)
0806 04:40:25.322982 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:25.322986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:25.323809 (+ 823us) server_negotiation.cc:407] Connection header received
0806 04:40:25.325223 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:25.325226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:25.325288 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:25.325459 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:25.326652 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.327262 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.328397 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.328675 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.328796 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:25.331602 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:25.331607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:25.331609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:25.331638 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:25.333461 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:25.333998 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:25.334005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:25.334009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:25.334108 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:25.334530 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:25.334534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:25.334536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:25.334784 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:25.334856 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:25.335676 (+ 820us) server_negotiation.cc:299] Negotiation successful
0806 04:40:25.335820 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 04:40:25.336995 11185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:25.323194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:44437 (local address 127.31.166.129:53461)
0806 04:40:25.323646 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:25.323689 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:25.325000 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:25.325631 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:25.325637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:25.326025 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:25.326493 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:25.326506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.327391 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.327396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:25.328255 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:25.328264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.328850 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.328855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:25.329032 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:25.329661 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:25.329677 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:25.331382 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:25.333587 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:25.333592 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:25.333605 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:25.333874 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:25.334217 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:25.334220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:25.334222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:25.334412 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:25.334940 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:25.334947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:25.335354 (+ 407us) client_negotiation.cc:769] Sending connection context
0806 04:40:25.335633 (+ 279us) client_negotiation.cc:240] Negotiation successful
0806 04:40:25.336018 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
I20260806 04:40:25.338296 11182 heartbeater.cc:347] Connected to a master server at 127.31.166.190:44437
I20260806 04:40:25.338565 11182 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:25.339133 11182 heartbeater.cc:511] Master 127.31.166.190:44437 requested a full tablet report, sending...
I20260806 04:40:25.340642 10975 ts_manager.cc:194] Registered new tserver with Master: 849a45448aaa4fbda6350eefaf14a41d (127.31.166.129:40901)
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
I20260806 04:40:25.341902 10975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:53461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:40:25.362175 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:44437
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:40:25.476406 11188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:25.476675 11188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:25.476718 11188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:25.480872 11188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:25.480937 11188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:25.481014 11188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:40:25.486531 11188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:44437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:25.487715 11188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:25.488767 11188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:40:25.499181 11188 server_base.cc:1034] running on GCE node
W20260806 04:40:25.499074 11196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:25.499069 11193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:25.499074 11194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:25.499821 11188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:25.500545 11188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:25.501709 11188 hybrid_clock.cc:648] HybridClock initialized: now 1785991225501682 us; error 39 us; skew 500 ppm
Aug 06 04:40:25 dist-test-slave-vkjp krb5kdc[10927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991225, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:25.504815 11188 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:40:25.505770 11188 webserver.cc:460] Webserver started at http://127.31.166.130:46099/ using document root <none> and password file <none>
I20260806 04:40:25.506363 11188 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:25.506458 11188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:25.506698 11188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:25.509016 11188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "ca7d2184c73b436fb9f582d91618c6fa"
format_stamp: "Formatted at 2026-08-06 04:40:25 on dist-test-slave-vkjp"
server_key: "5d91cd3cbdaefcf99bc240cdb4031d19"
server_key_iv: "b8e8790a65563e3bf09084c50779cd93"
server_key_version: "encryptionkey@0"
I20260806 04:40:25.509699 11188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "ca7d2184c73b436fb9f582d91618c6fa"
format_stamp: "Formatted at 2026-08-06 04:40:25 on dist-test-slave-vkjp"
server_key: "5d91cd3cbdaefcf99bc240cdb4031d19"
server_key_iv: "b8e8790a65563e3bf09084c50779cd93"
server_key_version: "encryptionkey@0"
I20260806 04:40:25.513573 11188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 04:40:25.516073 11203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:25.517441 11188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:40:25.517596 11188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "ca7d2184c73b436fb9f582d91618c6fa"
format_stamp: "Formatted at 2026-08-06 04:40:25 on dist-test-slave-vkjp"
server_key: "5d91cd3cbdaefcf99bc240cdb4031d19"
server_key_iv: "b8e8790a65563e3bf09084c50779cd93"
server_key_version: "encryptionkey@0"
I20260806 04:40:25.517720 11188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:25.537667 11188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:25.540843 11188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:25.541074 11188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:25.541684 11188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:25.542541 11188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:25.542613 11188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:25.542690 11188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:25.542743 11188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:25.552009 11188 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:40717
I20260806 04:40:25.552028 11326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:40717 every 8 connection(s)
I20260806 04:40:25.552983 11188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:40:25.559262 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 11188
I20260806 04:40:25.559405 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:40:25.559767 32410 external_mini_cluster.cc:1428] Setting key 77bbe7169784d6d3b1e86ae79e293733
Aug 06 04:40:25 dist-test-slave-vkjp krb5kdc[10927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991225, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:40:25.568593 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:25.554896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:38631 (local address 127.31.166.190:44437)
0806 04:40:25.555119 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:25.555126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:25.555972 (+ 846us) server_negotiation.cc:407] Connection header received
0806 04:40:25.557440 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:25.557444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:25.557503 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:25.557612 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:25.558797 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.559456 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.560979 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.561396 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.561512 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:25.564112 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:25.564119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:25.564125 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:25.564162 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:25.565936 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:25.566566 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:25.566569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:25.566571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:25.566666 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:25.567212 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:25.567215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:25.567216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:25.567435 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:25.567519 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:25.568312 (+ 793us) server_negotiation.cc:299] Negotiation successful
0806 04:40:25.568464 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:40:25.570143 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:25.555357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:44437 (local address 127.31.166.130:38631)
0806 04:40:25.555815 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:25.555860 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:25.557197 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:25.557781 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:25.557788 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:25.558140 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:25.558643 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:25.558657 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.559644 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.559647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:25.560833 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:25.560844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.561559 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.561562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:25.561711 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:25.562352 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:25.562375 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:25.563924 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:25.566071 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:25.566077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:25.566091 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:25.566393 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:25.566792 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:25.566797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:25.566801 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:25.567030 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:25.567565 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:25.567576 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:25.567991 (+ 415us) client_negotiation.cc:769] Sending connection context
0806 04:40:25.568240 (+ 249us) client_negotiation.cc:240] Negotiation successful
0806 04:40:25.568642 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:40:25.571511 11327 heartbeater.cc:347] Connected to a master server at 127.31.166.190:44437
I20260806 04:40:25.571779 11327 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:25.572337 11327 heartbeater.cc:511] Master 127.31.166.190:44437 requested a full tablet report, sending...
I20260806 04:40:25.573529 10975 ts_manager.cc:194] Registered new tserver with Master: ca7d2184c73b436fb9f582d91618c6fa (127.31.166.130:40717)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
I20260806 04:40:25.574191 10975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:38631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:40:25.591385 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:44437
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:37789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:40:25.703308 11332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:40:25.703601 11332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:40:25.703679 11332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:40:25.707759 11332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:40:25.707863 11332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:40:25.707959 11332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:40:25.713526 11332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:37789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:44437
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:40:25.714684 11332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:40:25.715795 11332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:40:25.725623 11337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:25.725634 11340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:40:25.725634 11338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:40:25.725740 11332 server_base.cc:1034] running on GCE node
I20260806 04:40:25.726342 11332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:40:25.727028 11332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:40:25.728248 11332 hybrid_clock.cc:648] HybridClock initialized: now 1785991225728210 us; error 43 us; skew 500 ppm
Aug 06 04:40:25 dist-test-slave-vkjp krb5kdc[10927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991225, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:40:25.731067 11332 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:40:25.732059 11332 webserver.cc:460] Webserver started at http://127.31.166.131:41899/ using document root <none> and password file <none>
I20260806 04:40:25.732625 11332 fs_manager.cc:359] Metadata directory not provided
I20260806 04:40:25.732718 11332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:40:25.732971 11332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:40:25.735172 11332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "5a63fc907c46468dadc08b8536ee5775"
format_stamp: "Formatted at 2026-08-06 04:40:25 on dist-test-slave-vkjp"
server_key: "823e489df9e3cae759d971fa88ab9600"
server_key_iv: "e507446cbcde8d53329f5222c77a85b8"
server_key_version: "encryptionkey@0"
I20260806 04:40:25.735924 11332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "5a63fc907c46468dadc08b8536ee5775"
format_stamp: "Formatted at 2026-08-06 04:40:25 on dist-test-slave-vkjp"
server_key: "823e489df9e3cae759d971fa88ab9600"
server_key_iv: "e507446cbcde8d53329f5222c77a85b8"
server_key_version: "encryptionkey@0"
I20260806 04:40:25.739889 11332 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260806 04:40:25.742461 11348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:25.743862 11332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:40:25.744009 11332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "5a63fc907c46468dadc08b8536ee5775"
format_stamp: "Formatted at 2026-08-06 04:40:25 on dist-test-slave-vkjp"
server_key: "823e489df9e3cae759d971fa88ab9600"
server_key_iv: "e507446cbcde8d53329f5222c77a85b8"
server_key_version: "encryptionkey@0"
I20260806 04:40:25.744131 11332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:40:25.760179 11332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:40:25.763643 11332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:40:25.763866 11332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:40:25.764478 11332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:40:25.765336 11332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:40:25.765407 11332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:25.765479 11332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:40:25.765530 11332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:40:25.774717 11332 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:46373
I20260806 04:40:25.774734 11472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:46373 every 8 connection(s)
I20260806 04:40:25.775728 11332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:40:25.778550 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 11332
I20260806 04:40:25.778698 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:40:25.779103 32410 external_mini_cluster.cc:1428] Setting key a81462b7d3c9e0cd73f35bd0a281bc2a
Aug 06 04:40:25 dist-test-slave-vkjp krb5kdc[10927](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991225, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:40:25.790309 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:25.777691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:45241 (local address 127.31.166.190:44437)
0806 04:40:25.777837 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:25.777841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:25.778730 (+ 889us) server_negotiation.cc:407] Connection header received
0806 04:40:25.780577 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:25.780580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:25.780638 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:25.780721 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:25.781990 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.782552 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.783784 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.784016 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.784131 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:25.786615 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:25.786621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:25.786623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:25.786650 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:25.788188 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:25.788723 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:25.788731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:25.788733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:25.788809 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:25.789172 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:25.789177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:25.789178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:25.789369 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:25.789479 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:25.789988 (+ 509us) server_negotiation.cc:299] Negotiation successful
0806 04:40:25.790134 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:40:25.791229 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:25.778002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:44437 (local address 127.31.166.131:45241)
0806 04:40:25.778544 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:25.778608 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:25.780345 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:25.780898 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:25.780905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:25.781290 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:40:25.781848 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:25.781862 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.782695 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.782698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:25.783655 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:25.783665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.784174 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.784178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:25.784344 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:25.784934 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:40:25.784951 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:40:25.786437 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:40:25.788314 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:25.788320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:25.788335 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:25.788607 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:25.788922 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:40:25.788925 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:40:25.788927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:40:25.789073 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:40:25.789478 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:40:25.789485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:40:25.789737 (+ 252us) client_negotiation.cc:769] Sending connection context
0806 04:40:25.789931 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 04:40:25.790267 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":113,"threads_started":1}
I20260806 04:40:25.792573 11473 heartbeater.cc:347] Connected to a master server at 127.31.166.190:44437
I20260806 04:40:25.792850 11473 heartbeater.cc:464] Registering TS with master...
I20260806 04:40:25.793411 11473 heartbeater.cc:511] Master 127.31.166.190:44437 requested a full tablet report, sending...
I20260806 04:40:25.794467 10974 ts_manager.cc:194] Registered new tserver with Master: 5a63fc907c46468dadc08b8536ee5775 (127.31.166.131:46373)
I20260806 04:40:25.795187 10974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:45241
I20260806 04:40:25.805110 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:40:25.814915 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:25.807099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56900 (local address 127.31.166.190:44437)
0806 04:40:25.807258 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:25.807263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:25.807409 (+ 146us) server_negotiation.cc:407] Connection header received
0806 04:40:25.807563 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:25.807569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:25.807620 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:25.807705 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:40:25.808436 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.808957 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.809910 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.810128 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.810233 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:25.811720 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:40:25.811723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:40:25.811726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:40:25.811758 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:40:25.813202 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:25.813655 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:25.813663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:25.813664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:25.813738 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:40:25.814016 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:40:25.814020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:40:25.814022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:40:25.814244 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:40:25.814384 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:25.814630 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 04:40:25.814742 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:40:25.818125 10974 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:40:25.821843 10974 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:40:25.836974 11489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:25.831687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38528 (local address 127.31.166.129:40901)
0806 04:40:25.832398 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:25.832411 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:25.832427 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:40:25.832484 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:25.832487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:25.832609 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:25.832761 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:40:25.833014 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.833733 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.835362 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.836214 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.836340 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:25.836473 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:25.836709 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 04:40:25.836815 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":132,"threads_started":1}
I20260806 04:40:25.836987 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:25.831265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:40901 (local address 127.0.0.1:38528)
0806 04:40:25.832092 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:25.832117 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:25.832201 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:25.832742 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:25.832745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:25.832763 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:40:25.832881 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:25.832891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.833872 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.833878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:25.835194 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:25.835205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.836390 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.836397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:25.836520 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:25.836558 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 04:40:25.836676 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 04:40:25.836797 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":4,"thread_start_us":86,"threads_started":1}
I20260806 04:40:25.837263 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:25.830660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.31.166.131:46373)
0806 04:40:25.831027 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:25.831040 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:25.831515 (+ 475us) server_negotiation.cc:407] Connection header received
0806 04:40:25.831815 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:25.831820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:25.832024 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:25.832154 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:40:25.832802 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.834011 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.835713 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.836366 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.836459 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:25.836598 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:25.836977 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 04:40:25.837089 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":166,"threads_started":1}
I20260806 04:40:25.837800 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:25.830472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:46373 (local address 127.0.0.1:46334)
0806 04:40:25.831358 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:25.831419 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:25.831633 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:25.832380 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:25.832386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:25.832419 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:40:25.832657 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:25.832665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.834200 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.834203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:25.835577 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:25.835588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.836729 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.836732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:25.836813 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:25.836832 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:40:25.837594 (+ 762us) client_negotiation.cc:240] Negotiation successful
0806 04:40:25.837672 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":209,"spinlock_wait_cycles":11008,"thread_start_us":92,"threads_started":1}
I20260806 04:40:25.838056 11486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:25.830713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51546 (local address 127.31.166.130:40717)
0806 04:40:25.831219 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:25.831231 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:25.831510 (+ 279us) server_negotiation.cc:407] Connection header received
0806 04:40:25.831815 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:25.831820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:25.832025 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:25.832154 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:40:25.832802 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.834010 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.835716 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.836517 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.836732 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:25.836850 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:25.837619 (+ 769us) server_negotiation.cc:299] Negotiation successful
0806 04:40:25.837725 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":111,"threads_started":1}
I20260806 04:40:25.838399 11487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:25.830472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:40717 (local address 127.0.0.1:51546)
0806 04:40:25.831358 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:25.831419 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:25.831633 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:25.832382 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:25.832386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:25.832418 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:40:25.832657 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:25.832665 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.834186 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.834191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:25.835535 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:25.835547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:25.837197 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:25.837203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:25.837291 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:25.837314 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:40:25.838197 (+ 883us) client_negotiation.cc:240] Negotiation successful
0806 04:40:25.838285 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":603,"spinlock_wait_cycles":10368,"thread_start_us":126,"threads_started":1}
I20260806 04:40:25.840270 11112 tablet_service.cc:1467] Processing CreateTablet for tablet af55f141b5554ff8849d6e3f61fa982c (DEFAULT_TABLE table=test-table [id=6af45d6cda5341d4a5df7d55a669a526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:25.841255 11112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af55f141b5554ff8849d6e3f61fa982c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:25.841120 11255 tablet_service.cc:1467] Processing CreateTablet for tablet af55f141b5554ff8849d6e3f61fa982c (DEFAULT_TABLE table=test-table [id=6af45d6cda5341d4a5df7d55a669a526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:25.841667 11399 tablet_service.cc:1467] Processing CreateTablet for tablet af55f141b5554ff8849d6e3f61fa982c (DEFAULT_TABLE table=test-table [id=6af45d6cda5341d4a5df7d55a669a526]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:40:25.842190 11255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af55f141b5554ff8849d6e3f61fa982c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:25.842903 11399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af55f141b5554ff8849d6e3f61fa982c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:40:25.847515 11490 tablet_bootstrap.cc:492] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d: Bootstrap starting.
I20260806 04:40:25.849565 11491 tablet_bootstrap.cc:492] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775: Bootstrap starting.
I20260806 04:40:25.850378 11492 tablet_bootstrap.cc:492] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa: Bootstrap starting.
I20260806 04:40:25.850914 11490 tablet_bootstrap.cc:654] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:25.851624 11491 tablet_bootstrap.cc:654] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:25.851799 11490 log.cc:824] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d: Log is configured to *not* fsync() on all Append() calls
I20260806 04:40:25.852639 11492 tablet_bootstrap.cc:654] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa: Neither blocks nor log segments found. Creating new log.
I20260806 04:40:25.852803 11491 log.cc:824] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775: Log is configured to *not* fsync() on all Append() calls
I20260806 04:40:25.853441 11492 log.cc:824] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa: Log is configured to *not* fsync() on all Append() calls
I20260806 04:40:25.854043 11490 tablet_bootstrap.cc:492] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d: No bootstrap required, opened a new log
I20260806 04:40:25.854238 11490 ts_tablet_manager.cc:1397] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 04:40:25.854758 11491 tablet_bootstrap.cc:492] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775: No bootstrap required, opened a new log
I20260806 04:40:25.854936 11491 ts_tablet_manager.cc:1397] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:40:25.855792 11492 tablet_bootstrap.cc:492] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa: No bootstrap required, opened a new log
I20260806 04:40:25.856010 11492 ts_tablet_manager.cc:1397] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:40:25.858788 11490 raft_consensus.cc:348] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a63fc907c46468dadc08b8536ee5775" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 46373 } } peers { permanent_uuid: "ca7d2184c73b436fb9f582d91618c6fa" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 40717 } } peers { permanent_uuid: "849a45448aaa4fbda6350eefaf14a41d" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40901 } }
I20260806 04:40:25.858788 11491 raft_consensus.cc:348] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a63fc907c46468dadc08b8536ee5775" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 46373 } } peers { permanent_uuid: "ca7d2184c73b436fb9f582d91618c6fa" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 40717 } } peers { permanent_uuid: "849a45448aaa4fbda6350eefaf14a41d" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40901 } }
I20260806 04:40:25.859057 11490 raft_consensus.cc:374] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:25.859057 11491 raft_consensus.cc:374] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:25.858923 11492 raft_consensus.cc:348] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a63fc907c46468dadc08b8536ee5775" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 46373 } } peers { permanent_uuid: "ca7d2184c73b436fb9f582d91618c6fa" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 40717 } } peers { permanent_uuid: "849a45448aaa4fbda6350eefaf14a41d" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40901 } }
I20260806 04:40:25.859134 11491 raft_consensus.cc:729] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a63fc907c46468dadc08b8536ee5775, State: Initialized, Role: FOLLOWER
I20260806 04:40:25.859134 11490 raft_consensus.cc:729] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849a45448aaa4fbda6350eefaf14a41d, State: Initialized, Role: FOLLOWER
I20260806 04:40:25.859189 11492 raft_consensus.cc:374] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:40:25.859241 11492 raft_consensus.cc:729] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca7d2184c73b436fb9f582d91618c6fa, State: Initialized, Role: FOLLOWER
I20260806 04:40:25.859635 11490 consensus_queue.cc:260] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a63fc907c46468dadc08b8536ee5775" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 46373 } } peers { permanent_uuid: "ca7d2184c73b436fb9f582d91618c6fa" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 40717 } } peers { permanent_uuid: "849a45448aaa4fbda6350eefaf14a41d" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40901 } }
I20260806 04:40:25.859623 11491 consensus_queue.cc:260] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a63fc907c46468dadc08b8536ee5775" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 46373 } } peers { permanent_uuid: "ca7d2184c73b436fb9f582d91618c6fa" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 40717 } } peers { permanent_uuid: "849a45448aaa4fbda6350eefaf14a41d" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40901 } }
I20260806 04:40:25.859695 11492 consensus_queue.cc:260] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a63fc907c46468dadc08b8536ee5775" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 46373 } } peers { permanent_uuid: "ca7d2184c73b436fb9f582d91618c6fa" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 40717 } } peers { permanent_uuid: "849a45448aaa4fbda6350eefaf14a41d" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40901 } }
I20260806 04:40:25.860561 11182 heartbeater.cc:503] Master 127.31.166.190:44437 was elected leader, sending a full tablet report...
I20260806 04:40:25.860617 11490 ts_tablet_manager.cc:1428] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:40:25.860616 11491 ts_tablet_manager.cc:1428] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260806 04:40:25.860698 11492 ts_tablet_manager.cc:1428] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 04:40:25.861049 11327 heartbeater.cc:503] Master 127.31.166.190:44437 was elected leader, sending a full tablet report...
I20260806 04:40:25.861207 11473 heartbeater.cc:503] Master 127.31.166.190:44437 was elected leader, sending a full tablet report...
W20260806 04:40:26.026778 11474 tablet.cc:2369] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:40:26.054255 11328 tablet.cc:2369] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:40:26.057462 11501 raft_consensus.cc:482] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:40:26.057700 11501 raft_consensus.cc:504] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a63fc907c46468dadc08b8536ee5775" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 46373 } } peers { permanent_uuid: "ca7d2184c73b436fb9f582d91618c6fa" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 40717 } } peers { permanent_uuid: "849a45448aaa4fbda6350eefaf14a41d" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40901 } }
I20260806 04:40:26.058930 11501 leader_election.cc:288] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a63fc907c46468dadc08b8536ee5775 (127.31.166.131:46373), 849a45448aaa4fbda6350eefaf14a41d (127.31.166.129:40901)
I20260806 04:40:26.063321 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:26.059447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:55581 (local address 127.31.166.131:46373)
0806 04:40:26.059591 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:26.059599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:26.059635 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:40:26.059802 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:26.059805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:26.059890 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:26.059968 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:40:26.060336 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:26.060978 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:26.062183 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:26.062665 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:26.062759 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:26.062800 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:26.063106 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 04:40:26.063199 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:40:26.063324 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:26.059384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:46373 (local address 127.31.166.130:55581)
0806 04:40:26.059549 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:26.059567 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:26.059684 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:26.060017 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:26.060020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:26.060047 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:40:26.060204 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:26.060212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:26.061114 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:26.061118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:26.062049 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:26.062059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:26.062789 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:26.062817 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:26.062941 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:26.062969 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 04:40:26.063053 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 04:40:26.063176 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":34}
I20260806 04:40:26.063428 11489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:26.059903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:56765 (local address 127.31.166.129:40901)
0806 04:40:26.060037 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:26.060042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:26.060167 (+ 125us) server_negotiation.cc:407] Connection header received
0806 04:40:26.060246 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:26.060252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:26.060325 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:26.060397 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:40:26.060672 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:26.061242 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:26.062386 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:26.062828 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:26.062949 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:26.062998 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:26.063215 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 04:40:26.063289 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:40:26.063385 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:26.059709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:40901 (local address 127.31.166.130:56765)
0806 04:40:26.060061 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:26.060076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:40:26.060170 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:40:26.060423 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:40:26.060426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:40:26.060443 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:40:26.060557 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:26.060564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:26.061389 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:26.061393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:26.062257 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:40:26.062270 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:40:26.062946 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:40:26.062948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:40:26.063056 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:26.063082 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:40:26.063185 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 04:40:26.063253 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":21,"thread_start_us":105,"threads_started":1}
I20260806 04:40:26.064250 11421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af55f141b5554ff8849d6e3f61fa982c" candidate_uuid: "ca7d2184c73b436fb9f582d91618c6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a63fc907c46468dadc08b8536ee5775" is_pre_election: true
I20260806 04:40:26.064249 11132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af55f141b5554ff8849d6e3f61fa982c" candidate_uuid: "ca7d2184c73b436fb9f582d91618c6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849a45448aaa4fbda6350eefaf14a41d" is_pre_election: true
I20260806 04:40:26.064595 11421 raft_consensus.cc:2454] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca7d2184c73b436fb9f582d91618c6fa in term 0.
I20260806 04:40:26.064595 11132 raft_consensus.cc:2454] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca7d2184c73b436fb9f582d91618c6fa in term 0.
I20260806 04:40:26.065058 11207 leader_election.cc:302] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a63fc907c46468dadc08b8536ee5775, ca7d2184c73b436fb9f582d91618c6fa; no voters:
I20260806 04:40:26.065356 11501 raft_consensus.cc:2781] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:40:26.065472 11501 raft_consensus.cc:482] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:40:26.065518 11501 raft_consensus.cc:3037] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:26.066740 11501 raft_consensus.cc:504] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a63fc907c46468dadc08b8536ee5775" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 46373 } } peers { permanent_uuid: "ca7d2184c73b436fb9f582d91618c6fa" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 40717 } } peers { permanent_uuid: "849a45448aaa4fbda6350eefaf14a41d" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40901 } }
I20260806 04:40:26.067159 11501 leader_election.cc:288] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [CANDIDATE]: Term 1 election: Requested vote from peers 5a63fc907c46468dadc08b8536ee5775 (127.31.166.131:46373), 849a45448aaa4fbda6350eefaf14a41d (127.31.166.129:40901)
I20260806 04:40:26.067504 11421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af55f141b5554ff8849d6e3f61fa982c" candidate_uuid: "ca7d2184c73b436fb9f582d91618c6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a63fc907c46468dadc08b8536ee5775"
I20260806 04:40:26.067570 11132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af55f141b5554ff8849d6e3f61fa982c" candidate_uuid: "ca7d2184c73b436fb9f582d91618c6fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849a45448aaa4fbda6350eefaf14a41d"
I20260806 04:40:26.067659 11421 raft_consensus.cc:3037] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:26.067703 11132 raft_consensus.cc:3037] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:40:26.068756 11132 raft_consensus.cc:2454] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca7d2184c73b436fb9f582d91618c6fa in term 1.
I20260806 04:40:26.068797 11421 raft_consensus.cc:2454] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca7d2184c73b436fb9f582d91618c6fa in term 1.
I20260806 04:40:26.069024 11207 leader_election.cc:302] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 849a45448aaa4fbda6350eefaf14a41d, ca7d2184c73b436fb9f582d91618c6fa; no voters:
I20260806 04:40:26.069190 11501 raft_consensus.cc:2781] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:40:26.069447 11501 raft_consensus.cc:686] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [term 1 LEADER]: Becoming Leader. State: Replica: ca7d2184c73b436fb9f582d91618c6fa, State: Running, Role: LEADER
I20260806 04:40:26.069795 11501 consensus_queue.cc:237] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a63fc907c46468dadc08b8536ee5775" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 46373 } } peers { permanent_uuid: "ca7d2184c73b436fb9f582d91618c6fa" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 40717 } } peers { permanent_uuid: "849a45448aaa4fbda6350eefaf14a41d" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40901 } }
W20260806 04:40:26.072422 11184 tablet.cc:2369] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:40:26.073118 10975 catalog_manager.cc:5495] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa reported cstate change: term changed from 0 to 1, leader changed from <none> to ca7d2184c73b436fb9f582d91618c6fa (127.31.166.130). New cstate: current_term: 1 leader_uuid: "ca7d2184c73b436fb9f582d91618c6fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a63fc907c46468dadc08b8536ee5775" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca7d2184c73b436fb9f582d91618c6fa" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 40717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849a45448aaa4fbda6350eefaf14a41d" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40901 } health_report { overall_health: UNKNOWN } } }
I20260806 04:40:26.132367 11486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:26.128726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.31.166.130:40717)
0806 04:40:26.128884 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 04:40:26.128887 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:40:26.128901 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:40:26.128977 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:40:26.128980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:40:26.129037 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:40:26.129133 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:40:26.129438 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:26.129967 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:26.130911 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:40:26.131146 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:40:26.131242 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:40:26.131559 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:40:26.131997 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:40:26.132092 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:40:26.132164 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 04:40:26.132218 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:40:26.138854 11132 raft_consensus.cc:1261] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d [term 1 FOLLOWER]: Refusing update from remote peer ca7d2184c73b436fb9f582d91618c6fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:40:26.138959 11421 raft_consensus.cc:1261] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775 [term 1 FOLLOWER]: Refusing update from remote peer ca7d2184c73b436fb9f582d91618c6fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:40:26.139729 11501 consensus_queue.cc:1035] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "849a45448aaa4fbda6350eefaf14a41d" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 40901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:40:26.139951 11503 consensus_queue.cc:1035] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a63fc907c46468dadc08b8536ee5775" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:40:26.149927 11507 mvcc.cc:205] Tried to move back new op lower bound from 7315420062256742400 to 7315420061984722944. Current Snapshot: MvccSnapshot[applied={T|T < 7315420062256742400 or (T in {7315420062256742400})}]
I20260806 04:40:26.158380 10973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:40:26.158542 10973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:40:26.161159 10973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e86166fa3a2401a8604513271cbab5b: Sending DeleteTablet for 3 replicas of tablet af55f141b5554ff8849d6e3f61fa982c
I20260806 04:40:26.161999 11255 tablet_service.cc:1514] Processing DeleteTablet for tablet af55f141b5554ff8849d6e3f61fa982c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:26 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:51546
I20260806 04:40:26.162071 11399 tablet_service.cc:1514] Processing DeleteTablet for tablet af55f141b5554ff8849d6e3f61fa982c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:26 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:46334
I20260806 04:40:26.162564 11517 tablet_replica.cc:330] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa: stopping tablet replica
I20260806 04:40:26.162566 11112 tablet_service.cc:1514] Processing DeleteTablet for tablet af55f141b5554ff8849d6e3f61fa982c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:40:26 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:38528
I20260806 04:40:26.163025 11517 raft_consensus.cc:2229] T af55f141b5554ff8849d6e3f61fa982c P ca7d2184c73b436fb9f582d91618c6fa [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:40:26.163050 11519 tablet_replica.cc:330] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d: stopping tablet replica
I20260806 04:40:26.163350 11519 raft_consensus.cc:2229] T af55f141b5554ff8849d6e3f61fa982c P 849a45448aaa4fbda6350eefaf14a41d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:40:26.163390 11518 tablet_replica.cc:330] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775: stopping tablet replica
I20260806 04:40:26.163511 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 11044
I20260806 04:40:26.166242 11518 raft_consensus.cc:2229] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:40:26.166596 11518 raft_consensus.cc:2258] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:40:26.168395 11518 ts_tablet_manager.cc:1905] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:40:26.171334 11518 ts_tablet_manager.cc:1918] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:40:26.171427 11518 log.cc:1196] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/af55f141b5554ff8849d6e3f61fa982c
I20260806 04:40:26.171762 11518 ts_tablet_manager.cc:1939] T af55f141b5554ff8849d6e3f61fa982c P 5a63fc907c46468dadc08b8536ee5775: Deleting consensus metadata
I20260806 04:40:26.173202 10960 catalog_manager.cc:4841] TS 5a63fc907c46468dadc08b8536ee5775 (127.31.166.131:46373): tablet af55f141b5554ff8849d6e3f61fa982c (table test-table [id=6af45d6cda5341d4a5df7d55a669a526]) successfully deleted
W20260806 04:40:26.177361 10960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:40901 (error 108)
I20260806 04:40:26.177451 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 11188
I20260806 04:40:26.179534 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:26.179164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:40901 (local address 127.0.0.1:38542)
0806 04:40:26.179360 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:26.179430 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:40901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260806 04:40:26.179700 10960 catalog_manager.cc:4584] TS 849a45448aaa4fbda6350eefaf14a41d (127.31.166.129:40901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af55f141b5554ff8849d6e3f61fa982c: Network error: Client connection negotiation failed: client connection to 127.31.166.129:40901: connect: Connection refused (error 111)
I20260806 04:40:26.184576 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 11332
I20260806 04:40:26.187183 11488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:40:26.184742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:40717 (local address 127.0.0.1:51574)
0806 04:40:26.186945 (+ 2203us) negotiation.cc:107] Waiting for socket to connect
0806 04:40:26.187020 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.130:40717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2097}
I20260806 04:40:26.191460 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 10941
2026-08-06T04:40:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2657 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:40:26 dist-test-slave-vkjp krb5kdc[11523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:40:26 dist-test-slave-vkjp krb5kdc[11523](info): set up 2 sockets
Aug 06 04:40:26 dist-test-slave-vkjp krb5kdc[11523](info): commencing operation
krb5kdc: starting...
W20260806 04:40:27.323390 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:40:27 dist-test-slave-vkjp krb5kdc[11523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:40:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:40:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.166.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.166.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/rangeradmin_127.31.166.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/rangeradmin_127.31.166.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.166.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.166.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/rangerlookup_127.31.166.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/rangerlookup_127.31.166.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.166.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.166.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/HTTP_127.31.166.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/HTTP_127.31.166.148@KRBTEST.COM.keytab.
I20260806 04:40:27.497974 32410 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskk51eOb/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/postgres -l logfile start
2026-08-06 04:40:30.211 UTC [11553] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 04:40:30.211 UTC [11553] LOG: listening on IPv4 address "127.31.166.148", port 42743
2026-08-06 04:40:30.215 UTC [11553] LOG: listening on Unix socket "/tmp/.s.PGSQL.42743"
2026-08-06 04:40:30.237 UTC [11556] LOG: database system was shut down at 2026-08-06 04:40:28 UTC
2026-08-06 04:40:30.241 UTC [11553] LOG: database system is ready to accept connections
I20260806 04:40:31.252426 32410 mini_postgres.cc:96] Postgres bound to 42743
2026-08-06 04:40:31.260 UTC [11564] FATAL: database "slave" does not exist
127.31.166.148:42743 - accepting connections
I20260806 04:40:31.261373 32410 mini_ranger.cc:161] Starting Ranger...
I20260806 04:40:31.278121 32410 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 04:40:31.802541 32410 mini_ranger.cc:87] Created ranger Postgres database
I20260806 04:40:31.802640 32410 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-admin
2026-08-06 04:40:32,126 [I] DB FLAVOR :POSTGRES
2026-08-06 04:40:32,126 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 04:40:32,127 [I] Checking connection..
2026-08-06 04:40:32,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 04:40:32,380 [I] Checking connection passed.
2026-08-06 04:40:32,380 [I] --------- Verifying version history table ---------
2026-08-06 04:40:32,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:40:32,640 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 04:40:32,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:40:32,895 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 04:40:32,895 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 04:40:32,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 04:40:33.125 UTC [11636] WARNING: there is no transaction in progress
2026-08-06 04:40:33,132 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 04:40:33,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:40:33,398 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 04:40:33,398 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 04:40:33,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 04:40:33,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 04:40:33,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vkjp.c.gcp-upstream.internal','N') ;"
2026-08-06 04:40:34,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 04:40:34,401 [I] Table x_portal_user does not exist in database ranger
2026-08-06 04:40:34,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 04:40:34,680 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 04:40:34,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 04:40:34,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 04:40:35,212 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 04:40:35,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 04:40:36.235 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.253 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.270 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.286 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.303 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.322 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.385 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.392 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.408 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.419 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.429 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.443 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.454 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.461 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.468 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:36.476 UTC [11758] WARNING: there is no transaction in progress
2026-08-06 04:40:37,177 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 04:40:37,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 04:40:37,430 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 04:40:37,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 04:40:37,688 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 04:40:37,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 04:40:37,968 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 04:40:37,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 04:40:38,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 04:40:38,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vkjp.c.gcp-upstream.internal';"
2026-08-06 04:40:38,816 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 04:40:38,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 04:40:39,086 [I] DB_PATCHES have already been applied
I20260806 04:40:39.092964 32410 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-admin:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskk51eOb/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 04:40:39.093039 32410 mini_ranger.cc:193] Using host: 127.31.166.148
I20260806 04:40:39.097142 32410 mini_ranger.cc:227] Ranger admin URL: http://127.31.166.148:46969
Aug 06, 2026 4:40:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:40:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:40:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 4:40:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 4:40:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 4:40:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 4:40:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 04:40:40.253983 32410 mini_ranger.cc:160] Time spent starting Ranger: real 8.993s user 0.003s sys 0.008s
Aug 06, 2026 4:41:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 4:41:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 4:41:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 4:41:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 4:41:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 4:41:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 4:41:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 4:41:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 4:41:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 4:41:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 4:41:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 4:41:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 4:41:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 4:41:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 4:41:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 4:41:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 4:41:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 4:41:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 04:41:03.215050 32410 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.166.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.166.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/rangerkms_127.31.166.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/rangerkms_127.31.166.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.166.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.166.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.166.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/HTTP_127.31.166.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.166.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/HTTP_127.31.166.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 04:41:03 dist-test-slave-vkjp krb5kdc[11523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991263, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 04:41:03.306402 32410 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 04:41:03.316465 32410 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 04:41:04.233430 32410 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 04:41:04.233547 32410 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-kms
I20260806 04:41:04.233669 32410 mini_ranger_kms.cc:227] Using postgres at 127.31.166.148:42743
2026-08-06 04:41:04,465 [I] DB FLAVOR :POSTGRES
2026-08-06 04:41:04,465 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 04:41:04,465 [I] Checking connection
2026-08-06 04:41:04,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 04:41:04,722 [I] connection success
2026-08-06 04:41:04,722 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 04:41:04,722 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 04:41:04,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 04:41:04,990 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 04:41:04,991 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 04:41:04,991 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 04:41:04,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:42743/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 04:41:05,254 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 04:41:05.767666 32410 mini_ranger_kms.cc:310] Created kms service
I20260806 04:41:05.966967 32410 mini_ranger_kms.cc:326] Created kudu user
I20260806 04:41:06.023383 32410 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 04:41:06 dist-test-slave-vkjp krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991263, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:41:06.464506 32410 mini_ranger_kms.cc:384] Added ranger policy
I20260806 04:41:06.464660 32410 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-kms:/tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 04:41:06.464681 32410 mini_ranger_kms.cc:240] Using host: 127.31.166.148
I20260806 04:41:06.468914 32410 mini_ranger_kms.cc:276] Ranger KMS PID: 11951
I20260806 04:41:06.469063 32410 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.166.148:42463
Aug 06, 2026 4:41:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:41:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:41:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 4:41:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 4:41:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 4:41:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 04:41:08.890035 32410 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.584s user 0.013s sys 0.203s
I20260806 04:41:08.890187 32410 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 04:41:08.890338 32410 mini_ranger_kms.cc:401] 127.31.166.148:42463/kms/v1/keys
Aug 06 04:41:09 dist-test-slave-vkjp krb5kdc[11523](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785991269, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.166.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 04:41:09 dist-test-slave-vkjp krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991269, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.166.148@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
Aug 06 04:41:09 dist-test-slave-vkjp krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991269, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.166.148@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
Aug 06 04:41:12 dist-test-slave-vkjp krb5kdc[11523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:41:12.155985 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:41915
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:41915
--ranger_config_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.166.148:42463/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:41:12.292725 12005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:41:12.293037 12005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:41:12.293125 12005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:41:12.297664 12005 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:41:12.297760 12005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:41:12.297798 12005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:41:12.297842 12005 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:41:12.297883 12005 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:41:12.304296 12005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.166.148:42463/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:41915
--ranger_config_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:41915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:41:12.305418 12005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:41:12.306509 12005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:41:12.316843 12012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:41:12.316818 12011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:41:12.317008 12005 server_base.cc:1034] running on GCE node
W20260806 04:41:12.316814 12014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:41:12.317912 12005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:41:12.319324 12005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:41:12.320530 12005 hybrid_clock.cc:648] HybridClock initialized: now 1785991272320499 us; error 40 us; skew 500 ppm
Aug 06 04:41:12 dist-test-slave-vkjp krb5kdc[11523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991272, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:41:12.324172 12005 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:41:12.325197 12005 webserver.cc:460] Webserver started at http://127.31.166.190:39825/ using document root <none> and password file <none>
I20260806 04:41:12.325775 12005 fs_manager.cc:359] Metadata directory not provided
I20260806 04:41:12.325838 12005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:41:12.326071 12005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:41:12 dist-test-slave-vkjp krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991272, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:41:12.382648 12005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "a9042046ceae4471aadf39ffca2bc899"
format_stamp: "Formatted at 2026-08-06 04:41:12 on dist-test-slave-vkjp"
server_key: "a709e6c3aa3d37945cd0ab5f87a6777b"
server_key_iv: "19e659e1496f6c854868a89da72812c8"
server_key_version: "kuduclusterkey@0"
I20260806 04:41:12.383334 12005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "a9042046ceae4471aadf39ffca2bc899"
format_stamp: "Formatted at 2026-08-06 04:41:12 on dist-test-slave-vkjp"
server_key: "a709e6c3aa3d37945cd0ab5f87a6777b"
server_key_iv: "19e659e1496f6c854868a89da72812c8"
server_key_version: "kuduclusterkey@0"
I20260806 04:41:12.387444 12005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260806 04:41:12.406427 12021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:41:12.408239 12005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:41:12.408402 12005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "a9042046ceae4471aadf39ffca2bc899"
format_stamp: "Formatted at 2026-08-06 04:41:12 on dist-test-slave-vkjp"
server_key: "a709e6c3aa3d37945cd0ab5f87a6777b"
server_key_iv: "19e659e1496f6c854868a89da72812c8"
server_key_version: "kuduclusterkey@0"
I20260806 04:41:12.408545 12005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:41:12.441262 12005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:41:12.444773 12005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:41:12.445011 12005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:41:12.454120 12005 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:41915
I20260806 04:41:12.454195 12075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:41915 every 8 connection(s)
I20260806 04:41:12.455334 12005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:41:12.455574 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 12005
I20260806 04:41:12.455726 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:41:12.456173 32410 external_mini_cluster.cc:1428] Setting key 8d23cce980171dbe76fa8175ad8c5d51
I20260806 04:41:12.459005 12076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 04:41:12 dist-test-slave-vkjp krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:41:12.466533 12076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899: Bootstrap starting.
I20260806 04:41:12.469375 12076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899: Neither blocks nor log segments found. Creating new log.
I20260806 04:41:12.470824 12076 log.cc:824] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899: Log is configured to *not* fsync() on all Append() calls
I20260806 04:41:12.471874 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:12.457787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.31.166.190:41915)
0806 04:41:12.458218 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 04:41:12.458237 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:41:12.458285 (+ 48us) server_negotiation.cc:407] Connection header received
0806 04:41:12.459056 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:41:12.459131 (+ 75us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:41:12.459501 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:41:12.459858 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:41:12.460807 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:12.462003 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:12.462972 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:12.463549 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:12.463727 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:12.466504 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:41:12.466509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:41:12.466522 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:41:12.466551 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:41:12.468507 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:41:12.468998 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:41:12.469008 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:41:12.469019 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:41:12.469136 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:41:12.469444 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:41:12.469450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:41:12.469452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:41:12.469907 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:41:12.470124 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 04:41:12.470446 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 04:41:12.470765 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":157,"threads_started":1}
I20260806 04:41:12.473372 12076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899: No bootstrap required, opened a new log
I20260806 04:41:12.477408 12076 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9042046ceae4471aadf39ffca2bc899" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 41915 } }
I20260806 04:41:12.477679 12076 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:41:12.477727 12076 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9042046ceae4471aadf39ffca2bc899, State: Initialized, Role: FOLLOWER
I20260806 04:41:12.478188 12076 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9042046ceae4471aadf39ffca2bc899" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 41915 } }
I20260806 04:41:12.478346 12076 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:41:12.478404 12076 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:41:12.478500 12076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:41:12.479673 12076 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9042046ceae4471aadf39ffca2bc899" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 41915 } }
I20260806 04:41:12.480036 12076 leader_election.cc:302] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9042046ceae4471aadf39ffca2bc899; no voters:
I20260806 04:41:12.480393 12076 leader_election.cc:288] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:41:12.480571 12081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:41:12.481194 12081 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [term 1 LEADER]: Becoming Leader. State: Replica: a9042046ceae4471aadf39ffca2bc899, State: Running, Role: LEADER
I20260806 04:41:12.481593 12081 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9042046ceae4471aadf39ffca2bc899" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 41915 } }
I20260806 04:41:12.481712 12076 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:41:12.484488 12083 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9042046ceae4471aadf39ffca2bc899. Latest consensus state: current_term: 1 leader_uuid: "a9042046ceae4471aadf39ffca2bc899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9042046ceae4471aadf39ffca2bc899" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 41915 } } }
I20260806 04:41:12.484633 12083 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [sys.catalog]: This master's current role is: LEADER
I20260806 04:41:12.484906 12082 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9042046ceae4471aadf39ffca2bc899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9042046ceae4471aadf39ffca2bc899" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 41915 } } }
I20260806 04:41:12.485000 12082 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899 [sys.catalog]: This master's current role is: LEADER
I20260806 04:41:12.485774 12092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:41:12.486078 12076 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 04:41:12.489037 12092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:41:12.503860 12092 catalog_manager.cc:1347] Generated new cluster ID: 3e727b3078be4360aabeb469ee331f53
I20260806 04:41:12.503957 12092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:41:12.514037 12092 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:41:12.515537 12092 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:41:12.522596 12092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899: Generated new TSK 0
I20260806 04:41:12.523574 12092 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:41:13.295245 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.166.148:42463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:41915
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:41:13.463040 12123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:41:13.463491 12123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:41:13.463598 12123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:41:13.469949 12123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:41:13.470052 12123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:41:13.470153 12123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:41:13.478529 12123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.166.148:42463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:41915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:41:13.480103 12123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:41:13.481561 12123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:41:13.494618 12130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:41:13.494580 12134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:41:13.494592 12131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:41:13.495947 12123 server_base.cc:1034] running on GCE node
I20260806 04:41:13.496472 12123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:41:13.497372 12123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:41:13.498602 12123 hybrid_clock.cc:648] HybridClock initialized: now 1785991273498573 us; error 47 us; skew 500 ppm
Aug 06 04:41:13 dist-test-slave-vkjp krb5kdc[11523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991273, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:41:13.502153 12123 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:41:13.503335 12123 webserver.cc:460] Webserver started at http://127.31.166.129:46687/ using document root <none> and password file <none>
I20260806 04:41:13.504009 12123 fs_manager.cc:359] Metadata directory not provided
I20260806 04:41:13.504105 12123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:41:13.504392 12123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:41:13 dist-test-slave-vkjp krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991273, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:41:13.525089 12123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "1c19a4fb5bbc41c4a90506148397eaae"
format_stamp: "Formatted at 2026-08-06 04:41:13 on dist-test-slave-vkjp"
server_key: "23ebf9674c12038b0d1c671fca3dfedd"
server_key_iv: "7f1c72159529b06c26545fe7c54b9459"
server_key_version: "kuduclusterkey@0"
I20260806 04:41:13.525802 12123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "1c19a4fb5bbc41c4a90506148397eaae"
format_stamp: "Formatted at 2026-08-06 04:41:13 on dist-test-slave-vkjp"
server_key: "23ebf9674c12038b0d1c671fca3dfedd"
server_key_iv: "7f1c72159529b06c26545fe7c54b9459"
server_key_version: "kuduclusterkey@0"
I20260806 04:41:13.530776 12123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 04:41:13.550009 12141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:41:13.551849 12123 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260806 04:41:13.552090 12123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "1c19a4fb5bbc41c4a90506148397eaae"
format_stamp: "Formatted at 2026-08-06 04:41:13 on dist-test-slave-vkjp"
server_key: "23ebf9674c12038b0d1c671fca3dfedd"
server_key_iv: "7f1c72159529b06c26545fe7c54b9459"
server_key_version: "kuduclusterkey@0"
I20260806 04:41:13.552333 12123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:41:13.574970 12123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:41:13.578380 12123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:41:13.578613 12123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:41:13.579358 12123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:41:13.580327 12123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:41:13.580394 12123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:41:13.580492 12123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:41:13.580529 12123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:41:13.596271 12123 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:34539
I20260806 04:41:13.596347 12264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:34539 every 8 connection(s)
I20260806 04:41:13.597808 12123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:41:13.607508 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 12123
I20260806 04:41:13.607681 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:41:13.608060 32410 external_mini_cluster.cc:1428] Setting key 09c1d34d663829a127364d35e017d4f7
Aug 06 04:41:13 dist-test-slave-vkjp krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991273, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:41:13.615797 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:13.601511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:37191 (local address 127.31.166.190:41915)
0806 04:41:13.601791 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 04:41:13.601796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:41:13.602620 (+ 824us) server_negotiation.cc:407] Connection header received
0806 04:41:13.603695 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:41:13.603699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:41:13.603762 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:41:13.603881 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:41:13.604990 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:13.605555 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:13.606720 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:13.606963 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:13.607079 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:13.610411 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:41:13.610415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:41:13.610418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:41:13.610445 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:41:13.612574 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:41:13.613354 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:41:13.613359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:41:13.613363 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:41:13.613466 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:41:13.614006 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:41:13.614012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:41:13.614017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:41:13.614335 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:41:13.614415 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:41:13.615367 (+ 952us) server_negotiation.cc:299] Negotiation successful
0806 04:41:13.615524 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":96,"threads_started":1}
I20260806 04:41:13.617983 12268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:13.602001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:41915 (local address 127.31.166.129:37191)
0806 04:41:13.602464 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0806 04:41:13.602517 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 04:41:13.603471 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:41:13.604060 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:41:13.604067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:41:13.604348 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:41:13.604837 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:13.604851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:13.605692 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:13.605695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:13.606584 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:13.606593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:13.607166 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:13.607171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:13.607401 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:13.608125 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:41:13.608149 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:41:13.610203 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:41:13.612738 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:41:13.612746 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:41:13.612765 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:41:13.613219 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:41:13.613589 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:41:13.613595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:41:13.613600 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:41:13.613848 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:41:13.614500 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:41:13.614509 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:41:13.614946 (+ 437us) client_negotiation.cc:769] Sending connection context
0806 04:41:13.616315 (+ 1369us) client_negotiation.cc:240] Negotiation successful
0806 04:41:13.616735 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":126,"threads_started":1}
I20260806 04:41:13.619822 12265 heartbeater.cc:347] Connected to a master server at 127.31.166.190:41915
I20260806 04:41:13.620087 12265 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
I20260806 04:41:13.620719 12265 heartbeater.cc:511] Master 127.31.166.190:41915 requested a full tablet report, sending...
I20260806 04:41:13.622442 12039 ts_manager.cc:194] Registered new tserver with Master: 1c19a4fb5bbc41c4a90506148397eaae (127.31.166.129:34539)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
I20260806 04:41:13.623885 12039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:37191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:41:13.643371 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.166.148:42463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:41915
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:41:13 dist-test-slave-vkjp krb5kdc[11523](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785991273, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 04:41:13.818676 12271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:41:13.819190 12271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:41:13.819315 12271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:41:13.825274 12271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:41:13.825416 12271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:41:13.825542 12271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:41:13.833129 12271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.166.148:42463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:41915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:41:13.834689 12271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:41:13.836143 12271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:41:13.848496 12278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:41:13.848589 12279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:41:13.848955 12271 server_base.cc:1034] running on GCE node
W20260806 04:41:13.851228 12282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:41:13.851751 12271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:41:13.852514 12271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:41:13.853696 12271 hybrid_clock.cc:648] HybridClock initialized: now 1785991273853665 us; error 48 us; skew 500 ppm
Aug 06 04:41:13 dist-test-slave-vkjp krb5kdc[11523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991273, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:41:13.856937 12271 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:41:13.857936 12271 webserver.cc:460] Webserver started at http://127.31.166.130:46335/ using document root <none> and password file <none>
I20260806 04:41:13.858593 12271 fs_manager.cc:359] Metadata directory not provided
I20260806 04:41:13.858688 12271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:41:13.858963 12271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:41:13 dist-test-slave-vkjp krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991273, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:41:13.873718 12271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "abfb13848e784fc9a7070c1be1f1e517"
format_stamp: "Formatted at 2026-08-06 04:41:13 on dist-test-slave-vkjp"
server_key: "93bde047984d1619c4c46f09aebbd1b6"
server_key_iv: "5dacf69b1ae128eaa1bd60708776eda7"
server_key_version: "kuduclusterkey@0"
I20260806 04:41:13.874341 12271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "abfb13848e784fc9a7070c1be1f1e517"
format_stamp: "Formatted at 2026-08-06 04:41:13 on dist-test-slave-vkjp"
server_key: "93bde047984d1619c4c46f09aebbd1b6"
server_key_iv: "5dacf69b1ae128eaa1bd60708776eda7"
server_key_version: "kuduclusterkey@0"
I20260806 04:41:13.880596 12271 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 04:41:13.912180 12290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:41:13.913844 12271 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:41:13.913995 12271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "abfb13848e784fc9a7070c1be1f1e517"
format_stamp: "Formatted at 2026-08-06 04:41:13 on dist-test-slave-vkjp"
server_key: "93bde047984d1619c4c46f09aebbd1b6"
server_key_iv: "5dacf69b1ae128eaa1bd60708776eda7"
server_key_version: "kuduclusterkey@0"
I20260806 04:41:13.914132 12271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:41:13.938115 12271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:41:13.942009 12271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:41:13.942296 12271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:41:13.943030 12271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:41:13.944177 12271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:41:13.944285 12271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:41:13.944386 12271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:41:13.944448 12271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:41:13.959350 12271 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:33777
I20260806 04:41:13.960660 12271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:41:13.961608 12410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:33777 every 8 connection(s)
I20260806 04:41:13.964186 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 12271
I20260806 04:41:13.964347 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:41:13.964706 32410 external_mini_cluster.cc:1428] Setting key b997ca6db2673c33eeee45238491fb9c
Aug 06 04:41:13 dist-test-slave-vkjp krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991273, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:41:13.980979 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:13.963900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:39039 (local address 127.31.166.190:41915)
0806 04:41:13.964042 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 04:41:13.964047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:41:13.967143 (+ 3096us) server_negotiation.cc:407] Connection header received
0806 04:41:13.968609 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:41:13.968613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:41:13.968666 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:41:13.968718 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:41:13.970562 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:13.971407 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:13.973099 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:13.973433 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:13.973505 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:13.976659 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:41:13.976663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:41:13.976665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:41:13.976694 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:41:13.978460 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:41:13.979126 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:41:13.979130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:41:13.979132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:41:13.979220 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:41:13.979619 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:41:13.979623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:41:13.979624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:41:13.979865 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:41:13.979949 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 04:41:13.980651 (+ 702us) server_negotiation.cc:299] Negotiation successful
0806 04:41:13.980792 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:41:13.983002 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:13.964400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:41915 (local address 127.31.166.130:39039)
0806 04:41:13.966944 (+ 2544us) negotiation.cc:107] Waiting for socket to connect
0806 04:41:13.967002 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 04:41:13.968118 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:41:13.969130 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:41:13.969139 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:41:13.969490 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:41:13.970371 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:13.970390 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:13.971579 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:13.971582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:13.972957 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:13.972968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:13.973714 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:13.973717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:13.973937 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:13.974660 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:41:13.974681 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:41:13.976456 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:41:13.978603 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:41:13.978610 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:41:13.978629 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:41:13.978983 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:41:13.979330 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:41:13.979333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:41:13.979335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:41:13.979513 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:41:13.980034 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:41:13.980041 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:41:13.980369 (+ 328us) client_negotiation.cc:769] Sending connection context
0806 04:41:13.981572 (+ 1203us) client_negotiation.cc:240] Negotiation successful
0806 04:41:13.981944 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2308,"thread_start_us":147,"threads_started":1}
I20260806 04:41:13.984786 12411 heartbeater.cc:347] Connected to a master server at 127.31.166.190:41915
I20260806 04:41:13.985033 12411 heartbeater.cc:464] Registering TS with master...
I20260806 04:41:13.985606 12411 heartbeater.cc:511] Master 127.31.166.190:41915 requested a full tablet report, sending...
I20260806 04:41:13.986878 12039 ts_manager.cc:194] Registered new tserver with Master: abfb13848e784fc9a7070c1be1f1e517 (127.31.166.130:33777)
I20260806 04:41:13.987790 12039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:39039
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:41:14.026479 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.166.148:42463/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:41915
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:38353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:41:14.208658 12417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:41:14.209044 12417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:41:14.209112 12417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:41:14.215384 12417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:41:14.215471 12417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:41:14.215572 12417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:41:14.223960 12417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:38353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.166.148:42463/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:41915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:41:14.225463 12417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:41:14.226869 12417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 06 04:41:14 dist-test-slave-vkjp krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991273, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
W20260806 04:41:14.240077 12428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:41:14.240653 12426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:41:14.241815 12424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:41:14.242080 12417 server_base.cc:1034] running on GCE node
I20260806 04:41:14.242673 12417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:41:14.243541 12417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:41:14.244791 12417 hybrid_clock.cc:648] HybridClock initialized: now 1785991274244676 us; error 81 us; skew 500 ppm
Aug 06 04:41:14 dist-test-slave-vkjp krb5kdc[11523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991274, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:41:14.251817 12417 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:41:14.253022 12417 webserver.cc:460] Webserver started at http://127.31.166.131:34567/ using document root <none> and password file <none>
I20260806 04:41:14.253733 12417 fs_manager.cc:359] Metadata directory not provided
I20260806 04:41:14.253824 12417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:41:14.254055 12417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:41:14 dist-test-slave-vkjp krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991274, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:41:14.268748 12417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "1f49495d2ca147bd8d27753c07673033"
format_stamp: "Formatted at 2026-08-06 04:41:14 on dist-test-slave-vkjp"
server_key: "d7ad26c9edc7021ed966a49596bc1e11"
server_key_iv: "fbbc1bfda9a7c573c6fb18df76cc6291"
server_key_version: "kuduclusterkey@0"
I20260806 04:41:14.269527 12417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "1f49495d2ca147bd8d27753c07673033"
format_stamp: "Formatted at 2026-08-06 04:41:14 on dist-test-slave-vkjp"
server_key: "d7ad26c9edc7021ed966a49596bc1e11"
server_key_iv: "fbbc1bfda9a7c573c6fb18df76cc6291"
server_key_version: "kuduclusterkey@0"
I20260806 04:41:14.277168 12417 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260806 04:41:14.314040 12435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:41:14.316004 12417 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 04:41:14.316128 12417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "1f49495d2ca147bd8d27753c07673033"
format_stamp: "Formatted at 2026-08-06 04:41:14 on dist-test-slave-vkjp"
server_key: "d7ad26c9edc7021ed966a49596bc1e11"
server_key_iv: "fbbc1bfda9a7c573c6fb18df76cc6291"
server_key_version: "kuduclusterkey@0"
I20260806 04:41:14.316253 12417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:41:14.336216 12417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:41:14.339903 12417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:41:14.340149 12417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:41:14.340822 12417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:41:14.341904 12417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:41:14.342017 12417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:41:14.342114 12417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:41:14.342178 12417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:41:14.353286 12417 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:38285
I20260806 04:41:14.353597 12555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:38285 every 8 connection(s)
I20260806 04:41:14.354383 12417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:41:14.355055 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 12417
I20260806 04:41:14.355216 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:41:14.355561 32410 external_mini_cluster.cc:1428] Setting key fd870ce3c7ed2834f34c8ebfbc96343b
Aug 06 04:41:14 dist-test-slave-vkjp krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991274, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:41:14.370652 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.357916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:56925 (local address 127.31.166.190:41915)
0806 04:41:14.358032 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 04:41:14.358036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:41:14.358051 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:41:14.358850 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:41:14.358854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:41:14.358905 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:41:14.358990 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:41:14.360365 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.361197 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.362741 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.363102 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.363202 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:14.366064 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:41:14.366068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:41:14.366071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:41:14.366097 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:41:14.368040 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:41:14.368741 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:41:14.368744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:41:14.368746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:41:14.368848 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:41:14.369286 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:41:14.369289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:41:14.369291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:41:14.369529 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:41:14.369604 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 04:41:14.370356 (+ 752us) server_negotiation.cc:299] Negotiation successful
0806 04:41:14.370491 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:41:14.372361 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.356883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:41915 (local address 127.31.166.131:56925)
0806 04:41:14.357452 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 04:41:14.357510 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 04:41:14.358615 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:41:14.359213 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:41:14.359221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:41:14.359564 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:41:14.360205 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:14.360222 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.361338 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.361341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:14.362608 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:14.362619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.363245 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.363249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:14.363451 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:14.364174 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:41:14.364195 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:41:14.365873 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:41:14.368182 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:41:14.368188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:41:14.368206 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:41:14.368624 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:41:14.368965 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:41:14.368968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:41:14.368970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:41:14.369168 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:41:14.369690 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:41:14.369697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:41:14.370056 (+ 359us) client_negotiation.cc:769] Sending connection context
0806 04:41:14.370837 (+ 781us) client_negotiation.cc:240] Negotiation successful
0806 04:41:14.371241 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":162,"threads_started":1}
I20260806 04:41:14.373494 12556 heartbeater.cc:347] Connected to a master server at 127.31.166.190:41915
I20260806 04:41:14.373737 12556 heartbeater.cc:464] Registering TS with master...
I20260806 04:41:14.374294 12556 heartbeater.cc:511] Master 127.31.166.190:41915 requested a full tablet report, sending...
I20260806 04:41:14.375407 12039 ts_manager.cc:194] Registered new tserver with Master: 1f49495d2ca147bd8d27753c07673033 (127.31.166.131:38285)
I20260806 04:41:14.376124 12039 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:56925
I20260806 04:41:14.381426 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:41:14.400460 12267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.387563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.31.166.190:41915)
0806 04:41:14.387685 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 04:41:14.387689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:41:14.388030 (+ 341us) server_negotiation.cc:407] Connection header received
0806 04:41:14.388142 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:41:14.388146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:41:14.388196 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:41:14.388278 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:41:14.392150 (+ 3872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.392987 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.394238 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.394580 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.394684 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:14.396539 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:41:14.396543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:41:14.396547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:41:14.396582 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:41:14.398383 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:41:14.398923 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:41:14.398927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:41:14.398929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:41:14.399031 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:41:14.399438 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:41:14.399441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:41:14.399443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:41:14.399663 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:41:14.399766 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 04:41:14.400139 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 04:41:14.400266 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:41:14.405490 12039 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:41:14.409633 12039 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:41:14.428532 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.420652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:33777 (local address 127.0.0.1:56768)
0806 04:41:14.421443 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0806 04:41:14.421493 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:41:14.421637 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:41:14.422454 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:41:14.422460 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:41:14.422485 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:41:14.423469 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:14.423478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.424926 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.424929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:14.426451 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:14.426463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.427974 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.427977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:14.428090 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:14.428135 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 04:41:14.428247 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 04:41:14.428368 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":108,"threads_started":1}
I20260806 04:41:14.429098 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.420112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56768 (local address 127.31.166.130:33777)
0806 04:41:14.421174 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0806 04:41:14.421191 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:41:14.421792 (+ 601us) server_negotiation.cc:407] Connection header received
0806 04:41:14.421878 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:41:14.421883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:41:14.422102 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:41:14.422262 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:41:14.423620 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.424796 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.426618 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.427609 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.427739 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:14.427875 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 04:41:14.428846 (+ 971us) server_negotiation.cc:299] Negotiation successful
0806 04:41:14.428955 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":949,"thread_start_us":166,"threads_started":1}
I20260806 04:41:14.432228 12339 tablet_service.cc:1467] Processing CreateTablet for tablet c056296d62b44977863eaefd096fb399 (DEFAULT_TABLE table=test-table [id=b7d7a06a915140c6bf815bf58ea9dc52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:41:14.433415 12339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c056296d62b44977863eaefd096fb399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:41:14.440227 12571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.421874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:38285 (local address 127.0.0.1:35028)
0806 04:41:14.425036 (+ 3162us) negotiation.cc:107] Waiting for socket to connect
0806 04:41:14.425050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:41:14.425127 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:41:14.427124 (+ 1997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:41:14.427127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:41:14.427146 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:41:14.427248 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:14.427255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.432711 (+ 5456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.432714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:14.434878 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:14.434890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.439725 (+ 4835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.439729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:14.439837 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:14.439869 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 04:41:14.439951 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 04:41:14.440046 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3094,"thread_start_us":90,"threads_started":1}
I20260806 04:41:14.441233 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.423055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:34539 (local address 127.0.0.1:59662)
0806 04:41:14.427485 (+ 4430us) negotiation.cc:107] Waiting for socket to connect
0806 04:41:14.427502 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:41:14.427687 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:41:14.428454 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:41:14.428458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:41:14.428478 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:41:14.428563 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:14.428569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.435024 (+ 6455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.435028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:14.436212 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:14.436224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.440796 (+ 4572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.440801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:14.440892 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:14.440919 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 04:41:14.441008 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 04:41:14.441092 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4368,"thread_start_us":587,"threads_started":1}
I20260806 04:41:14.441908 12573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.423313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59662 (local address 127.31.166.129:34539)
0806 04:41:14.426222 (+ 2909us) server_negotiation.cc:206] Beginning negotiation
0806 04:41:14.426229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:41:14.427608 (+ 1379us) server_negotiation.cc:407] Connection header received
0806 04:41:14.427850 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:41:14.427857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:41:14.428059 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:41:14.428233 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:41:14.430018 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.431192 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.436393 (+ 5201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.437250 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.437406 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:14.441572 (+ 4166us) server_negotiation.cc:1035] Waiting for connection context
0806 04:41:14.441638 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 04:41:14.441758 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1940,"thread_start_us":99,"threads_started":1}
I20260806 04:41:14.442622 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.422290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.31.166.131:38285)
0806 04:41:14.426435 (+ 4145us) server_negotiation.cc:206] Beginning negotiation
0806 04:41:14.426447 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:41:14.426473 (+ 26us) server_negotiation.cc:407] Connection header received
0806 04:41:14.426551 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:41:14.426557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:41:14.426769 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:41:14.426935 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:41:14.431372 (+ 4437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.432562 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.438439 (+ 5877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.439358 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.439502 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:14.439646 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 04:41:14.442354 (+ 2708us) server_negotiation.cc:299] Negotiation successful
0806 04:41:14.442470 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4046,"thread_start_us":161,"threads_started":1}
I20260806 04:41:14.445618 12178 tablet_service.cc:1467] Processing CreateTablet for tablet c056296d62b44977863eaefd096fb399 (DEFAULT_TABLE table=test-table [id=b7d7a06a915140c6bf815bf58ea9dc52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:41:14.445781 12472 tablet_service.cc:1467] Processing CreateTablet for tablet c056296d62b44977863eaefd096fb399 (DEFAULT_TABLE table=test-table [id=b7d7a06a915140c6bf815bf58ea9dc52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:41:14.446754 12178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c056296d62b44977863eaefd096fb399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:41:14.446796 12472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c056296d62b44977863eaefd096fb399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:41:14.456542 12575 tablet_bootstrap.cc:492] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517: Bootstrap starting.
I20260806 04:41:14.457691 12576 tablet_bootstrap.cc:492] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033: Bootstrap starting.
I20260806 04:41:14.461632 12576 tablet_bootstrap.cc:654] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033: Neither blocks nor log segments found. Creating new log.
I20260806 04:41:14.462574 12576 log.cc:824] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033: Log is configured to *not* fsync() on all Append() calls
I20260806 04:41:14.465260 12575 tablet_bootstrap.cc:654] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517: Neither blocks nor log segments found. Creating new log.
I20260806 04:41:14.466159 12575 log.cc:824] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517: Log is configured to *not* fsync() on all Append() calls
I20260806 04:41:14.469523 12576 tablet_bootstrap.cc:492] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033: No bootstrap required, opened a new log
I20260806 04:41:14.469741 12576 ts_tablet_manager.cc:1397] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260806 04:41:14.471487 12579 tablet_bootstrap.cc:492] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae: Bootstrap starting.
I20260806 04:41:14.473862 12579 tablet_bootstrap.cc:654] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae: Neither blocks nor log segments found. Creating new log.
I20260806 04:41:14.473994 12576 raft_consensus.cc:348] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfb13848e784fc9a7070c1be1f1e517" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 33777 } } peers { permanent_uuid: "1f49495d2ca147bd8d27753c07673033" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38285 } } peers { permanent_uuid: "1c19a4fb5bbc41c4a90506148397eaae" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34539 } }
I20260806 04:41:14.474283 12576 raft_consensus.cc:374] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:41:14.474359 12576 raft_consensus.cc:729] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f49495d2ca147bd8d27753c07673033, State: Initialized, Role: FOLLOWER
I20260806 04:41:14.475564 12579 log.cc:824] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae: Log is configured to *not* fsync() on all Append() calls
I20260806 04:41:14.476013 12575 tablet_bootstrap.cc:492] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517: No bootstrap required, opened a new log
I20260806 04:41:14.476215 12575 ts_tablet_manager.cc:1397] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260806 04:41:14.479450 12576 consensus_queue.cc:260] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfb13848e784fc9a7070c1be1f1e517" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 33777 } } peers { permanent_uuid: "1f49495d2ca147bd8d27753c07673033" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38285 } } peers { permanent_uuid: "1c19a4fb5bbc41c4a90506148397eaae" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34539 } }
I20260806 04:41:14.480366 12575 raft_consensus.cc:348] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfb13848e784fc9a7070c1be1f1e517" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 33777 } } peers { permanent_uuid: "1f49495d2ca147bd8d27753c07673033" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38285 } } peers { permanent_uuid: "1c19a4fb5bbc41c4a90506148397eaae" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34539 } }
I20260806 04:41:14.480616 12575 raft_consensus.cc:374] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:41:14.480557 12576 ts_tablet_manager.cc:1428] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260806 04:41:14.480669 12575 raft_consensus.cc:729] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abfb13848e784fc9a7070c1be1f1e517, State: Initialized, Role: FOLLOWER
I20260806 04:41:14.481046 12556 heartbeater.cc:503] Master 127.31.166.190:41915 was elected leader, sending a full tablet report...
I20260806 04:41:14.481129 12575 consensus_queue.cc:260] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfb13848e784fc9a7070c1be1f1e517" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 33777 } } peers { permanent_uuid: "1f49495d2ca147bd8d27753c07673033" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38285 } } peers { permanent_uuid: "1c19a4fb5bbc41c4a90506148397eaae" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34539 } }
I20260806 04:41:14.482072 12575 ts_tablet_manager.cc:1428] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:41:14.482429 12411 heartbeater.cc:503] Master 127.31.166.190:41915 was elected leader, sending a full tablet report...
I20260806 04:41:14.485870 12579 tablet_bootstrap.cc:492] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae: No bootstrap required, opened a new log
I20260806 04:41:14.486085 12579 ts_tablet_manager.cc:1397] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260806 04:41:14.491652 12579 raft_consensus.cc:348] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfb13848e784fc9a7070c1be1f1e517" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 33777 } } peers { permanent_uuid: "1f49495d2ca147bd8d27753c07673033" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38285 } } peers { permanent_uuid: "1c19a4fb5bbc41c4a90506148397eaae" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34539 } }
I20260806 04:41:14.492071 12579 raft_consensus.cc:374] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:41:14.492187 12579 raft_consensus.cc:729] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c19a4fb5bbc41c4a90506148397eaae, State: Initialized, Role: FOLLOWER
I20260806 04:41:14.492867 12579 consensus_queue.cc:260] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfb13848e784fc9a7070c1be1f1e517" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 33777 } } peers { permanent_uuid: "1f49495d2ca147bd8d27753c07673033" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38285 } } peers { permanent_uuid: "1c19a4fb5bbc41c4a90506148397eaae" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34539 } }
I20260806 04:41:14.494174 12579 ts_tablet_manager.cc:1428] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 04:41:14.494593 12265 heartbeater.cc:503] Master 127.31.166.190:41915 was elected leader, sending a full tablet report...
Aug 06 04:41:14 dist-test-slave-vkjp krb5kdc[11523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991273, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
W20260806 04:41:14.601552 12266 tablet.cc:2369] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:41:14.608655 12557 tablet.cc:2369] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:41:14.659428 12582 raft_consensus.cc:482] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:41:14.659685 12582 raft_consensus.cc:504] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfb13848e784fc9a7070c1be1f1e517" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 33777 } } peers { permanent_uuid: "1f49495d2ca147bd8d27753c07673033" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38285 } } peers { permanent_uuid: "1c19a4fb5bbc41c4a90506148397eaae" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34539 } }
I20260806 04:41:14.661175 12582 leader_election.cc:288] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abfb13848e784fc9a7070c1be1f1e517 (127.31.166.130:33777), 1c19a4fb5bbc41c4a90506148397eaae (127.31.166.129:34539)
I20260806 04:41:14.667392 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.661664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:33777 (local address 127.31.166.131:52407)
0806 04:41:14.662543 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0806 04:41:14.662559 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:41:14.662659 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:41:14.663453 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:41:14.663456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:41:14.663478 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:41:14.663615 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:14.663622 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.664719 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.664722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:14.665946 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:14.665957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.666952 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.666956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:14.667063 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:14.667106 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 04:41:14.667177 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 04:41:14.667248 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789}
I20260806 04:41:14.667881 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.662920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:52407 (local address 127.31.166.130:33777)
0806 04:41:14.663068 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:41:14.663073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:41:14.663124 (+ 51us) server_negotiation.cc:407] Connection header received
0806 04:41:14.663187 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:41:14.663190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:41:14.663239 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:41:14.663320 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:41:14.663786 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.664581 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.666107 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.666715 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.666832 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:14.666891 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 04:41:14.667686 (+ 795us) server_negotiation.cc:299] Negotiation successful
0806 04:41:14.667768 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:41:14.668169 12573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.663066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:40105 (local address 127.31.166.129:34539)
0806 04:41:14.663223 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 04:41:14.663228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:41:14.663523 (+ 295us) server_negotiation.cc:407] Connection header received
0806 04:41:14.663724 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:41:14.663727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:41:14.663782 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:41:14.663835 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:41:14.664263 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.665000 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.666496 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.667134 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.667233 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:14.667282 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 04:41:14.667954 (+ 672us) server_negotiation.cc:299] Negotiation successful
0806 04:41:14.668042 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 04:41:14.668924 12345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c056296d62b44977863eaefd096fb399" candidate_uuid: "1f49495d2ca147bd8d27753c07673033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abfb13848e784fc9a7070c1be1f1e517" is_pre_election: true
I20260806 04:41:14.669210 12587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.661945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:34539 (local address 127.31.166.131:40105)
0806 04:41:14.663434 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0806 04:41:14.663449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:41:14.663602 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:41:14.663976 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:41:14.663978 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:41:14.663993 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:41:14.664121 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:14.664131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.665152 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.665155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:14.666340 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:41:14.666352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.667422 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.667426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:41:14.667791 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:14.667818 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 04:41:14.668851 (+ 1033us) client_negotiation.cc:240] Negotiation successful
0806 04:41:14.668932 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1416,"thread_start_us":153,"threads_started":1}
I20260806 04:41:14.669265 12345 raft_consensus.cc:2454] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f49495d2ca147bd8d27753c07673033 in term 0.
I20260806 04:41:14.670059 12437 leader_election.cc:302] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f49495d2ca147bd8d27753c07673033, abfb13848e784fc9a7070c1be1f1e517; no voters:
I20260806 04:41:14.670114 12203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c056296d62b44977863eaefd096fb399" candidate_uuid: "1f49495d2ca147bd8d27753c07673033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c19a4fb5bbc41c4a90506148397eaae" is_pre_election: true
I20260806 04:41:14.670364 12203 raft_consensus.cc:2454] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f49495d2ca147bd8d27753c07673033 in term 0.
I20260806 04:41:14.670401 12582 raft_consensus.cc:2781] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:41:14.670487 12582 raft_consensus.cc:482] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:41:14.670533 12582 raft_consensus.cc:3037] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:41:14.672165 12582 raft_consensus.cc:504] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfb13848e784fc9a7070c1be1f1e517" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 33777 } } peers { permanent_uuid: "1f49495d2ca147bd8d27753c07673033" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38285 } } peers { permanent_uuid: "1c19a4fb5bbc41c4a90506148397eaae" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34539 } }
I20260806 04:41:14.672590 12582 leader_election.cc:288] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [CANDIDATE]: Term 1 election: Requested vote from peers abfb13848e784fc9a7070c1be1f1e517 (127.31.166.130:33777), 1c19a4fb5bbc41c4a90506148397eaae (127.31.166.129:34539)
I20260806 04:41:14.673678 12345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c056296d62b44977863eaefd096fb399" candidate_uuid: "1f49495d2ca147bd8d27753c07673033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abfb13848e784fc9a7070c1be1f1e517"
I20260806 04:41:14.673812 12345 raft_consensus.cc:3037] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:41:14.675325 12345 raft_consensus.cc:2454] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f49495d2ca147bd8d27753c07673033 in term 1.
I20260806 04:41:14.675850 12203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c056296d62b44977863eaefd096fb399" candidate_uuid: "1f49495d2ca147bd8d27753c07673033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c19a4fb5bbc41c4a90506148397eaae"
I20260806 04:41:14.675982 12203 raft_consensus.cc:3037] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:41:14.677255 12203 raft_consensus.cc:2454] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f49495d2ca147bd8d27753c07673033 in term 1.
I20260806 04:41:14.677727 12437 leader_election.cc:302] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c19a4fb5bbc41c4a90506148397eaae, 1f49495d2ca147bd8d27753c07673033; no voters:
I20260806 04:41:14.678072 12582 raft_consensus.cc:2781] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:41:14.678335 12582 raft_consensus.cc:686] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [term 1 LEADER]: Becoming Leader. State: Replica: 1f49495d2ca147bd8d27753c07673033, State: Running, Role: LEADER
I20260806 04:41:14.678706 12582 consensus_queue.cc:237] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfb13848e784fc9a7070c1be1f1e517" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 33777 } } peers { permanent_uuid: "1f49495d2ca147bd8d27753c07673033" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38285 } } peers { permanent_uuid: "1c19a4fb5bbc41c4a90506148397eaae" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34539 } }
I20260806 04:41:14.682528 12038 catalog_manager.cc:5495] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f49495d2ca147bd8d27753c07673033 (127.31.166.131). New cstate: current_term: 1 leader_uuid: "1f49495d2ca147bd8d27753c07673033" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "abfb13848e784fc9a7070c1be1f1e517" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 33777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f49495d2ca147bd8d27753c07673033" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c19a4fb5bbc41c4a90506148397eaae" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34539 } health_report { overall_health: UNKNOWN } } }
W20260806 04:41:14.712917 12412 tablet.cc:2369] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:41:14.732318 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.727257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.31.166.131:38285)
0806 04:41:14.727409 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 04:41:14.727413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:41:14.727433 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:41:14.727524 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:41:14.727527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:41:14.727604 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:41:14.727737 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:41:14.728072 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.728748 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.730010 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:41:14.730354 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:41:14.730495 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:41:14.730872 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:41:14.731487 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:41:14.731611 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 04:41:14.732083 (+ 472us) server_negotiation.cc:299] Negotiation successful
0806 04:41:14.732153 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:41:14.740833 12345 raft_consensus.cc:1261] T c056296d62b44977863eaefd096fb399 P abfb13848e784fc9a7070c1be1f1e517 [term 1 FOLLOWER]: Refusing update from remote peer 1f49495d2ca147bd8d27753c07673033: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:41:14.740833 12203 raft_consensus.cc:1261] T c056296d62b44977863eaefd096fb399 P 1c19a4fb5bbc41c4a90506148397eaae [term 1 FOLLOWER]: Refusing update from remote peer 1f49495d2ca147bd8d27753c07673033: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:41:14.741655 12588 consensus_queue.cc:1035] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abfb13848e784fc9a7070c1be1f1e517" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 33777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:41:14.742148 12588 consensus_queue.cc:1035] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c19a4fb5bbc41c4a90506148397eaae" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:41:14.760921 12592 mvcc.cc:205] Tried to move back new op lower bound from 7315420261329833984 to 7315420261087178752. Current Snapshot: MvccSnapshot[applied={T|T < 7315420261329833984}]
I20260806 04:41:14.769202 12038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:41:14.769400 12038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:41:14.771886 12038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9042046ceae4471aadf39ffca2bc899: Sending DeleteTablet for 3 replicas of tablet c056296d62b44977863eaefd096fb399
I20260806 04:41:14.772713 12472 tablet_service.cc:1514] Processing DeleteTablet for tablet c056296d62b44977863eaefd096fb399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:41:14 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:35028
I20260806 04:41:14.772809 12339 tablet_service.cc:1514] Processing DeleteTablet for tablet c056296d62b44977863eaefd096fb399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:41:14 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:56768
I20260806 04:41:14.773298 12600 tablet_replica.cc:330] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033: stopping tablet replica
I20260806 04:41:14.773684 12600 raft_consensus.cc:2229] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:41:14.773698 12178 tablet_service.cc:1514] Processing DeleteTablet for tablet c056296d62b44977863eaefd096fb399 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:41:14 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:59662
I20260806 04:41:14.774293 12600 raft_consensus.cc:2258] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:41:14.775249 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 12123
I20260806 04:41:14.777537 12600 ts_tablet_manager.cc:1905] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:41:14.781417 12600 ts_tablet_manager.cc:1918] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:41:14.781651 12600 log.cc:1196] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/c056296d62b44977863eaefd096fb399
I20260806 04:41:14.782209 12600 ts_tablet_manager.cc:1939] T c056296d62b44977863eaefd096fb399 P 1f49495d2ca147bd8d27753c07673033: Deleting consensus metadata
I20260806 04:41:14.783903 12022 catalog_manager.cc:4841] TS 1f49495d2ca147bd8d27753c07673033 (127.31.166.131:38285): tablet c056296d62b44977863eaefd096fb399 (table test-table [id=b7d7a06a915140c6bf815bf58ea9dc52]) successfully deleted
W20260806 04:41:14.786048 12023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:34539 (error 108)
I20260806 04:41:14.786984 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 12271
I20260806 04:41:14.787381 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.786859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:34539 (local address 127.0.0.1:59670)
0806 04:41:14.787072 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 04:41:14.787269 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:34539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260806 04:41:14.788061 12023 catalog_manager.cc:4584] TS 1c19a4fb5bbc41c4a90506148397eaae (127.31.166.129:34539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c056296d62b44977863eaefd096fb399: Network error: Client connection negotiation failed: client connection to 127.31.166.129:34539: connect: Connection refused (error 111)
I20260806 04:41:14.795280 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 12417
I20260806 04:41:14.798243 12570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:41:14.797840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:33777 (local address 127.0.0.1:56772)
0806 04:41:14.798025 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0806 04:41:14.798115 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.130:33777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260806 04:41:14.805480 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 12005
I20260806 04:41:14.812397 32410 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 04:41:15.147740 32410 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 04:41:15.147874 32410 mini_ranger.cc:66] Stopping Ranger...
I20260806 04:41:15.493330 32410 mini_ranger.cc:68] Stopped Ranger
2026-08-06 04:41:15.493 UTC [11553] LOG: received smart shutdown request
2026-08-06 04:41:15.497 UTC [11553] LOG: background worker "logical replication launcher" (PID 11562) exited with exit code 1
2026-08-06 04:41:15.498 UTC [11557] LOG: shutting down
2026-08-06 04:41:15.696 UTC [11553] LOG: database system is shut down
2026-08-06T04:41:15Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49626 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:41:15 dist-test-slave-vkjp krb5kdc[12614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:41:15 dist-test-slave-vkjp krb5kdc[12614](info): set up 2 sockets
Aug 06 04:41:15 dist-test-slave-vkjp krb5kdc[12614](info): commencing operation
krb5kdc: starting...
W20260806 04:41:16.917745 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:41:17 dist-test-slave-vkjp krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:41:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:41:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.166.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.166.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/rangeradmin_127.31.166.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/rangeradmin_127.31.166.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.166.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.166.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/rangerlookup_127.31.166.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/rangerlookup_127.31.166.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.166.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.166.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/HTTP_127.31.166.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/HTTP_127.31.166.148@KRBTEST.COM.keytab.
I20260806 04:41:17.101553 32410 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskk51eOb/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/postgres -l logfile start
2026-08-06 04:41:19.765 UTC [12645] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 04:41:19.765 UTC [12645] LOG: listening on IPv4 address "127.31.166.148", port 50955
2026-08-06 04:41:19.769 UTC [12645] LOG: listening on Unix socket "/tmp/.s.PGSQL.50955"
2026-08-06 04:41:19.790 UTC [12648] LOG: database system was shut down at 2026-08-06 04:41:18 UTC
2026-08-06 04:41:19.794 UTC [12645] LOG: database system is ready to accept connections
I20260806 04:41:20.886978 32410 mini_postgres.cc:96] Postgres bound to 50955
2026-08-06 04:41:20.893 UTC [12657] FATAL: database "slave" does not exist
127.31.166.148:50955 - accepting connections
I20260806 04:41:20.894100 32410 mini_ranger.cc:161] Starting Ranger...
I20260806 04:41:20.910167 32410 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 04:41:21.437629 32410 mini_ranger.cc:87] Created ranger Postgres database
I20260806 04:41:21.437731 32410 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-admin
2026-08-06 04:41:21,735 [I] DB FLAVOR :POSTGRES
2026-08-06 04:41:21,736 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 04:41:21,736 [I] Checking connection..
2026-08-06 04:41:21,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 04:41:21,992 [I] Checking connection passed.
2026-08-06 04:41:21,993 [I] --------- Verifying version history table ---------
2026-08-06 04:41:21,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:41:22,237 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 04:41:22,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:41:22,484 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 04:41:22,484 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 04:41:22,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 04:41:22.744 UTC [12725] WARNING: there is no transaction in progress
2026-08-06 04:41:22,751 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 04:41:22,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 04:41:23,026 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 04:41:23,026 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 04:41:23,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 04:41:23,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 04:41:23,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-vkjp.c.gcp-upstream.internal','N') ;"
2026-08-06 04:41:23,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 04:41:24,079 [I] Table x_portal_user does not exist in database ranger
2026-08-06 04:41:24,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 04:41:24,346 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 04:41:24,347 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 04:41:24,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 04:41:24,871 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 04:41:24,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 04:41:25.929 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:25.946 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:25.964 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:25.980 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:25.998 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:26.017 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:26.076 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:26.084 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:26.100 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:26.111 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:26.122 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:26.137 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:26.148 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:26.155 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:26.162 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:26.169 UTC [12841] WARNING: there is no transaction in progress
2026-08-06 04:41:26,852 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 04:41:26,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 04:41:27,107 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 04:41:27,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 04:41:27,387 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 04:41:27,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 04:41:27,644 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 04:41:27,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 04:41:27,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 04:41:28,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-vkjp.c.gcp-upstream.internal';"
2026-08-06 04:41:28,466 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 04:41:28,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 04:41:28,718 [I] DB_PATCHES have already been applied
I20260806 04:41:28.724934 32410 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-admin:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskk51eOb/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 04:41:28.725009 32410 mini_ranger.cc:193] Using host: 127.31.166.148
I20260806 04:41:28.728909 32410 mini_ranger.cc:227] Ranger admin URL: http://127.31.166.148:45611
Aug 06, 2026 4:41:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:41:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:41:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 4:41:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 4:41:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 4:41:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 4:41:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 04:41:29.913089 32410 mini_ranger.cc:160] Time spent starting Ranger: real 9.019s user 0.000s sys 0.010s
Aug 06, 2026 4:41:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 4:41:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 4:41:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 4:41:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 4:41:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 4:41:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 4:41:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 4:41:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 4:41:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 4:41:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 4:41:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 4:41:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 4:41:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 4:41:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 4:41:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 4:41:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 4:41:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 4:41:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 04:41:52.827999 32410 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.166.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.166.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/rangerkms_127.31.166.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.166.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/rangerkms_127.31.166.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.166.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.166.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.166.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/HTTP_127.31.166.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.166.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/HTTP_127.31.166.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 04:41:52 dist-test-slave-vkjp krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991312, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 04:41:52.920387 32410 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 04:41:52.931010 32410 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 04:41:53.987022 32410 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 04:41:53.987166 32410 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-kms
I20260806 04:41:53.987197 32410 mini_ranger_kms.cc:227] Using postgres at 127.31.166.148:50955
2026-08-06 04:41:54,182 [I] DB FLAVOR :POSTGRES
2026-08-06 04:41:54,183 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 04:41:54,183 [I] Checking connection
2026-08-06 04:41:54,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 04:41:54,428 [I] connection success
2026-08-06 04:41:54,428 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 04:41:54,429 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 04:41:54,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 04:41:54,685 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 04:41:54,685 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 04:41:54,685 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 04:41:54,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.166.148:50955/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 04:41:54,993 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 04:41:55.490620 32410 mini_ranger_kms.cc:310] Created kms service
I20260806 04:41:55.712050 32410 mini_ranger_kms.cc:326] Created kudu user
I20260806 04:41:55.759061 32410 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 04:41:55 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991312, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:41:56.219283 32410 mini_ranger_kms.cc:384] Added ranger policy
I20260806 04:41:56.219439 32410 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-kms:/tmp/dist-test-taskk51eOb/build/debug/bin/postgresql.jar:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskk51eOb/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 04:41:56.219458 32410 mini_ranger_kms.cc:240] Using host: 127.31.166.148
I20260806 04:41:56.223420 32410 mini_ranger_kms.cc:276] Ranger KMS PID: 13033
I20260806 04:41:56.223501 32410 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.166.148:58929
Aug 06, 2026 4:41:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 4:41:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 4:41:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 4:41:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 4:41:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 4:41:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260806 04:41:58.767601 32410 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.847s user 0.012s sys 0.166s
I20260806 04:41:58.767745 32410 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 04:41:58.767843 32410 mini_ranger_kms.cc:401] 127.31.166.148:58929/kms/v1/keys
Aug 06 04:41:58 dist-test-slave-vkjp krb5kdc[12614](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785991318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.166.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 06 04:41:59 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.166.148@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
Aug 06 04:41:59 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991318, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.166.148@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
Aug 06 04:42:01 dist-test-slave-vkjp krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:42:02.024040 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:36261
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:36261
--ranger_config_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.166.148:58929/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:02.143314 13086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:02.143615 13086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:02.143698 13086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:02.147919 13086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:42:02.147996 13086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:02.148017 13086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:02.148033 13086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:42:02.148049 13086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:42:02.153894 13086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.166.148:58929/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:36261
--ranger_config_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:36261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:02.155017 13086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:02.156052 13086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:02.165354 13094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:02.165369 13092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:02.165460 13086 server_base.cc:1034] running on GCE node
W20260806 04:42:02.165360 13091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:02.166268 13086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:02.167629 13086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:02.168823 13086 hybrid_clock.cc:648] HybridClock initialized: now 1785991322168799 us; error 38 us; skew 500 ppm
Aug 06 04:42:02 dist-test-slave-vkjp krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991322, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:02.171911 13086 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:42:02.172976 13086 webserver.cc:460] Webserver started at http://127.31.166.190:38019/ using document root <none> and password file <none>
I20260806 04:42:02.173566 13086 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:02.173650 13086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:02.173897 13086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:42:02 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991322, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:42:02.232968 13086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "a21945364f0e48c2a4a226ad95cee176"
format_stamp: "Formatted at 2026-08-06 04:42:02 on dist-test-slave-vkjp"
server_key: "9e2bda51fc49263442a3e41cc0ca6af6"
server_key_iv: "68895ecd91034bf73691b333d22b6390"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:02.233654 13086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "a21945364f0e48c2a4a226ad95cee176"
format_stamp: "Formatted at 2026-08-06 04:42:02 on dist-test-slave-vkjp"
server_key: "9e2bda51fc49263442a3e41cc0ca6af6"
server_key_iv: "68895ecd91034bf73691b333d22b6390"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:02.237922 13086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:42:02.254696 13101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:02.256265 13086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:42:02.256439 13086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "a21945364f0e48c2a4a226ad95cee176"
format_stamp: "Formatted at 2026-08-06 04:42:02 on dist-test-slave-vkjp"
server_key: "9e2bda51fc49263442a3e41cc0ca6af6"
server_key_iv: "68895ecd91034bf73691b333d22b6390"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:02.256605 13086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:02.278193 13086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:02.281455 13086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:02.281699 13086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:02.290328 13086 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:36261
I20260806 04:42:02.290330 13156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:36261 every 8 connection(s)
I20260806 04:42:02.291476 13086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:42:02.293179 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 13086
I20260806 04:42:02.293327 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:42:02.293648 32410 external_mini_cluster.cc:1428] Setting key b401f07bd6630c1e6889ce36eae040dc
I20260806 04:42:02.295341 13157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 04:42:02 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:02.303923 13157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176: Bootstrap starting.
I20260806 04:42:02.307140 13157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:02.308822 13157 log.cc:824] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:02.309384 13160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:02.295242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50144 (local address 127.31.166.190:36261)
0806 04:42:02.295721 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:02.295742 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:02.295787 (+ 45us) server_negotiation.cc:407] Connection header received
0806 04:42:02.296433 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:02.296449 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:02.296792 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:02.297142 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:02.298179 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:02.299361 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:02.300328 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:02.300836 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:02.301032 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:02.304232 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:02.304237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:02.304252 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:02.304290 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:02.306037 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:02.306535 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:02.306541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:02.306545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:02.306660 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:02.306975 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:02.306979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:02.306981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:02.307544 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:02.307755 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:02.308067 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 04:42:02.308500 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":170,"threads_started":1}
I20260806 04:42:02.311460 13157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176: No bootstrap required, opened a new log
I20260806 04:42:02.315806 13157 raft_consensus.cc:348] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21945364f0e48c2a4a226ad95cee176" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36261 } }
I20260806 04:42:02.316149 13157 raft_consensus.cc:374] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:02.316213 13157 raft_consensus.cc:729] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a21945364f0e48c2a4a226ad95cee176, State: Initialized, Role: FOLLOWER
I20260806 04:42:02.316725 13157 consensus_queue.cc:260] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21945364f0e48c2a4a226ad95cee176" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36261 } }
I20260806 04:42:02.316890 13157 raft_consensus.cc:388] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:42:02.316955 13157 raft_consensus.cc:482] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:42:02.317057 13157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:02.318439 13157 raft_consensus.cc:504] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21945364f0e48c2a4a226ad95cee176" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36261 } }
I20260806 04:42:02.318837 13157 leader_election.cc:302] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a21945364f0e48c2a4a226ad95cee176; no voters:
I20260806 04:42:02.319281 13157 leader_election.cc:288] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:42:02.319398 13162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:42:02.319692 13162 raft_consensus.cc:686] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 1 LEADER]: Becoming Leader. State: Replica: a21945364f0e48c2a4a226ad95cee176, State: Running, Role: LEADER
I20260806 04:42:02.320086 13162 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21945364f0e48c2a4a226ad95cee176" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36261 } }
I20260806 04:42:02.320734 13157 sys_catalog.cc:564] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:42:02.321358 13162 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21945364f0e48c2a4a226ad95cee176. Latest consensus state: current_term: 1 leader_uuid: "a21945364f0e48c2a4a226ad95cee176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21945364f0e48c2a4a226ad95cee176" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36261 } } }
I20260806 04:42:02.321580 13162 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:02.321803 13163 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a21945364f0e48c2a4a226ad95cee176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21945364f0e48c2a4a226ad95cee176" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36261 } } }
I20260806 04:42:02.321955 13163 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:02.322249 13171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:42:02.325673 13171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:42:02.326263 13157 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 04:42:02.331894 13171 catalog_manager.cc:1347] Generated new cluster ID: c8298f05cf72442eb3c052a596076e60
I20260806 04:42:02.331969 13171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:42:02.351749 13171 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:42:02.352802 13171 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:42:02.365403 13171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176: Generated new TSK 0
I20260806 04:42:02.366218 13171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:42:03.059418 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.166.148:58929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36261
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:03.195166 13202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:03.195595 13202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:03.195668 13202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:03.202121 13202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:03.202198 13202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:03.202304 13202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:42:03.208216 13202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.166.148:58929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:03.209358 13202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:03.210415 13202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:03.221108 13209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:03.221127 13211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:03.221465 13208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:03.221586 13202 server_base.cc:1034] running on GCE node
I20260806 04:42:03.222121 13202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:03.222908 13202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:03.224093 13202 hybrid_clock.cc:648] HybridClock initialized: now 1785991323224083 us; error 57 us; skew 500 ppm
Aug 06 04:42:03 dist-test-slave-vkjp krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991323, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:03.227187 13202 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:42:03.228240 13202 webserver.cc:460] Webserver started at http://127.31.166.129:39779/ using document root <none> and password file <none>
I20260806 04:42:03.228878 13202 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:03.228974 13202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:03.229244 13202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:42:03 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991323, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:42:03.245976 13202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "40ffb574abdd448a9cfdc4cb5e52d934"
format_stamp: "Formatted at 2026-08-06 04:42:03 on dist-test-slave-vkjp"
server_key: "b2362f351099245ee5b007e656fb8541"
server_key_iv: "61737e419491e800c70b69c8a58a230b"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:03.246799 13202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "40ffb574abdd448a9cfdc4cb5e52d934"
format_stamp: "Formatted at 2026-08-06 04:42:03 on dist-test-slave-vkjp"
server_key: "b2362f351099245ee5b007e656fb8541"
server_key_iv: "61737e419491e800c70b69c8a58a230b"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:03.252228 13202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 04:42:03.271387 13220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:03.273162 13202 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 04:42:03.273361 13202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "40ffb574abdd448a9cfdc4cb5e52d934"
format_stamp: "Formatted at 2026-08-06 04:42:03 on dist-test-slave-vkjp"
server_key: "b2362f351099245ee5b007e656fb8541"
server_key_iv: "61737e419491e800c70b69c8a58a230b"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:03.273564 13202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:03.288112 13202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:03.292098 13202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:03.292351 13202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:03.293112 13202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:03.294226 13202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:03.294349 13202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:03.294513 13202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:03.294610 13202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:03.306500 13202 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:35417
I20260806 04:42:03.306557 13343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:35417 every 8 connection(s)
I20260806 04:42:03.307721 13202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:42:03.317952 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 13202
I20260806 04:42:03.318123 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:42:03.318449 32410 external_mini_cluster.cc:1428] Setting key 981c051f3ab30e74cf9a2dcc7cd1af6b
Aug 06 04:42:03 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991323, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:03.330580 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:03.309702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:47311 (local address 127.31.166.190:36261)
0806 04:42:03.309959 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:03.309966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:03.310851 (+ 885us) server_negotiation.cc:407] Connection header received
0806 04:42:03.312050 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:03.312054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:03.312115 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:03.312183 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:03.313852 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:03.314753 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:03.316542 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:03.316918 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:03.317037 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:03.325665 (+ 8628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:03.325670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:03.325673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:03.325706 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:03.327818 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:03.328578 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:03.328581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:03.328583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:03.328679 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:03.329110 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:03.329114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:03.329115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:03.329349 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:03.329441 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:03.330211 (+ 770us) server_negotiation.cc:299] Negotiation successful
0806 04:42:03.330363 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
I20260806 04:42:03.332645 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:03.310201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36261 (local address 127.31.166.129:47311)
0806 04:42:03.310707 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:03.310750 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:03.311779 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:03.312597 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:03.312605 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:03.312972 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:03.313665 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:03.313687 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:03.314948 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:03.314952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:03.316391 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:03.316403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:03.317229 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:03.317233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:03.317466 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:03.323448 (+ 5982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:03.323476 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:03.325388 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:03.328000 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:03.328007 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:03.328027 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:03.328446 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:03.328795 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:03.328798 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:03.328800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:03.328994 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:03.329511 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:03.329518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:03.329864 (+ 346us) client_negotiation.cc:769] Sending connection context
0806 04:42:03.331136 (+ 1272us) client_negotiation.cc:240] Negotiation successful
0806 04:42:03.331540 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":161,"threads_started":1}
I20260806 04:42:03.334306 13344 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36261
I20260806 04:42:03.334578 13344 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
I20260806 04:42:03.337000 13344 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
I20260806 04:42:03.338963 13120 ts_manager.cc:194] Registered new tserver with Master: 40ffb574abdd448a9cfdc4cb5e52d934 (127.31.166.129:35417)
I20260806 04:42:03.340552 13120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:47311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:42:03.366686 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.166.148:58929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36261
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:42:03 dist-test-slave-vkjp krb5kdc[12614](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785991323, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 04:42:03.526037 13352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:03.526441 13352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:03.526510 13352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:03.532914 13352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:03.533012 13352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:03.533102 13352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:42:03.541391 13352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.166.148:58929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:03.543144 13352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:03.544545 13352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:03.556171 13361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:03.556172 13358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:03.556759 13359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:03.557641 13352 server_base.cc:1034] running on GCE node
I20260806 04:42:03.558072 13352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:03.558820 13352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:03.560070 13352 hybrid_clock.cc:648] HybridClock initialized: now 1785991323560035 us; error 51 us; skew 500 ppm
Aug 06 04:42:03 dist-test-slave-vkjp krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991323, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:03.563144 13352 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:42:03.564221 13352 webserver.cc:460] Webserver started at http://127.31.166.130:43565/ using document root <none> and password file <none>
I20260806 04:42:03.564867 13352 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:03.564962 13352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:03.565186 13352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:42:03 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991323, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:42:03.583307 13352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "ecf753b9ee334eec80d5aaaea61f4e8e"
format_stamp: "Formatted at 2026-08-06 04:42:03 on dist-test-slave-vkjp"
server_key: "e82d3e5042f7d9c4c8778e7aabeef11c"
server_key_iv: "49d022ac4ab6ed9dd2283eab56b4860a"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:03.584300 13352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "ecf753b9ee334eec80d5aaaea61f4e8e"
format_stamp: "Formatted at 2026-08-06 04:42:03 on dist-test-slave-vkjp"
server_key: "e82d3e5042f7d9c4c8778e7aabeef11c"
server_key_iv: "49d022ac4ab6ed9dd2283eab56b4860a"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:03.604803 13352 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.003s sys 0.001s
I20260806 04:42:03.623493 13368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:03.625336 13352 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260806 04:42:03.625574 13352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "ecf753b9ee334eec80d5aaaea61f4e8e"
format_stamp: "Formatted at 2026-08-06 04:42:03 on dist-test-slave-vkjp"
server_key: "e82d3e5042f7d9c4c8778e7aabeef11c"
server_key_iv: "49d022ac4ab6ed9dd2283eab56b4860a"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:03.625797 13352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:03.646200 13352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:03.650519 13352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:03.650743 13352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:03.651495 13352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:03.652433 13352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:03.652503 13352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:03.652598 13352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:03.652642 13352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:03.663753 13352 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:38717
I20260806 04:42:03.663961 13487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:38717 every 8 connection(s)
I20260806 04:42:03.664783 13352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:42:03.667961 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 13352
I20260806 04:42:03.668111 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:42:03.668385 32410 external_mini_cluster.cc:1428] Setting key c207147a68ddf3eee25da45081c4db36
Aug 06 04:42:03 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991323, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:03.681319 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:03.666715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:56811 (local address 127.31.166.190:36261)
0806 04:42:03.666855 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:03.666860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:03.669025 (+ 2165us) server_negotiation.cc:407] Connection header received
0806 04:42:03.670233 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:03.670236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:03.670296 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:03.670351 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:03.672132 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:03.673007 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:03.674260 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:03.674499 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:03.674613 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:03.677016 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:03.677019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:03.677021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:03.677045 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:03.678504 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:03.679113 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:03.679129 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:03.679134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:03.679229 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:03.679750 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:03.679757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:03.679761 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:03.680025 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:03.680114 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:03.680875 (+ 761us) server_negotiation.cc:299] Negotiation successful
0806 04:42:03.681059 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:42:03.682940 13490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:03.667226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36261 (local address 127.31.166.130:56811)
0806 04:42:03.668878 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:03.668939 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:03.669987 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:03.670783 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:03.670794 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:03.671198 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:03.671934 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:03.671958 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:03.673193 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:03.673197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:03.674117 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:03.674127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:03.674659 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:03.674663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:03.674837 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:03.675483 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:03.675500 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:03.676842 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:03.678684 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:03.678690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:03.678710 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:03.678979 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:03.679339 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:03.679345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:03.679350 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:03.679627 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:03.680186 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:03.680196 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:03.680558 (+ 362us) client_negotiation.cc:769] Sending connection context
0806 04:42:03.681874 (+ 1316us) client_negotiation.cc:240] Negotiation successful
0806 04:42:03.682154 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":168,"threads_started":1}
I20260806 04:42:03.684572 13488 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36261
I20260806 04:42:03.684836 13488 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:03.685493 13488 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
I20260806 04:42:03.686791 13120 ts_manager.cc:194] Registered new tserver with Master: ecf753b9ee334eec80d5aaaea61f4e8e (127.31.166.130:38717)
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:42:03.687664 13120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:56811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:42:03.710790 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.166.148:58929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36261
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:03.868216 13494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:03.868654 13494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:03.868769 13494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:03.875121 13494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:03.875216 13494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:03.875308 13494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:42:03.883455 13494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.166.148:58929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:03.884918 13494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:03.886296 13494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:03.899286 13502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:03.899497 13500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:03.899587 13494 server_base.cc:1034] running on GCE node
W20260806 04:42:03.899427 13499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:03.900377 13494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:03.901194 13494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:03.902407 13494 hybrid_clock.cc:648] HybridClock initialized: now 1785991323902384 us; error 37 us; skew 500 ppm
Aug 06 04:42:03 dist-test-slave-vkjp krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991323, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:03.906322 13494 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:42:03.907637 13494 webserver.cc:460] Webserver started at http://127.31.166.131:41671/ using document root <none> and password file <none>
I20260806 04:42:03.908459 13494 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:03.908622 13494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:03.908931 13494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 04:42:03 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991323, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:42:03.935225 13494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "075953f9b3a348f08ed2701343a5a136"
format_stamp: "Formatted at 2026-08-06 04:42:03 on dist-test-slave-vkjp"
server_key: "b0cb8741568eec352e1d30ad622c2c16"
server_key_iv: "92f06f944c420f2c51b9d0562a0d75ab"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:03.936306 13494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "075953f9b3a348f08ed2701343a5a136"
format_stamp: "Formatted at 2026-08-06 04:42:03 on dist-test-slave-vkjp"
server_key: "b0cb8741568eec352e1d30ad622c2c16"
server_key_iv: "92f06f944c420f2c51b9d0562a0d75ab"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:03.949152 13494 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.004s sys 0.000s
Aug 06 04:42:03 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991323, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:42:03.982048 13510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.001s user 0.000s sys 0.000s
I20260806 04:42:03.991168 13494 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.004s sys 0.001s
I20260806 04:42:03.991361 13494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "075953f9b3a348f08ed2701343a5a136"
format_stamp: "Formatted at 2026-08-06 04:42:03 on dist-test-slave-vkjp"
server_key: "b0cb8741568eec352e1d30ad622c2c16"
server_key_iv: "92f06f944c420f2c51b9d0562a0d75ab"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:03.991523 13494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:04.013553 13494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:04.017443 13494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:04.017700 13494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:04.018647 13494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:04.019826 13494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:04.019940 13494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:04.020047 13494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:04.020109 13494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:04.033249 13494 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:38491
I20260806 04:42:04.034684 13494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:42:04.036129 13631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:38491 every 8 connection(s)
I20260806 04:42:04.041440 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 13494
I20260806 04:42:04.041568 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:42:04.041863 32410 external_mini_cluster.cc:1428] Setting key 9ae1ad6b7ca4c61f04371a874806063c
Aug 06 04:42:04 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991323, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:04.061683 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:04.040757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:33347 (local address 127.31.166.190:36261)
0806 04:42:04.040890 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:04.040894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:04.045590 (+ 4696us) server_negotiation.cc:407] Connection header received
0806 04:42:04.046863 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:04.046867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:04.046925 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:04.047009 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:04.048623 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:04.049380 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:04.051066 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:04.051421 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:04.051503 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:04.056755 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:04.056759 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:04.056762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:04.056812 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:04.058682 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:04.059458 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:04.059462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:04.059464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:04.059556 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:04.060105 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:04.060108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:04.060110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:04.060351 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:04.060438 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:04.061314 (+ 876us) server_negotiation.cc:299] Negotiation successful
0806 04:42:04.061466 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:42:04.063760 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:04.039653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36261 (local address 127.31.166.131:33347)
0806 04:42:04.045397 (+ 5744us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:04.045469 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:04.046581 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:04.047373 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:04.047383 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:04.047750 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:04.048452 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:04.048472 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:04.049545 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:04.049548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:04.050914 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:04.050925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:04.051726 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:04.051731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:04.051948 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:04.052720 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:04.052742 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:04.056540 (+ 3798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:04.058841 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:04.058849 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:04.058869 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:04.059325 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:04.059677 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:04.059680 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:04.059682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:04.059955 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:04.060508 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:04.060515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:04.060879 (+ 364us) client_negotiation.cc:769] Sending connection context
0806 04:42:04.062199 (+ 1320us) client_negotiation.cc:240] Negotiation successful
0806 04:42:04.062592 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5474,"thread_start_us":149,"threads_started":1}
I20260806 04:42:04.065433 13632 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36261
I20260806 04:42:04.065761 13632 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:04.068363 13632 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
I20260806 04:42:04.069581 13120 ts_manager.cc:194] Registered new tserver with Master: 075953f9b3a348f08ed2701343a5a136 (127.31.166.131:38491)
I20260806 04:42:04.070329 13120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:33347
I20260806 04:42:04.079986 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:42:04.080859 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 13202
I20260806 04:42:04.091339 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 13352
I20260806 04:42:04.114888 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 13494
I20260806 04:42:04.124092 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 13086
I20260806 04:42:04.137097 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:36261
--webserver_interface=127.31.166.190
--webserver_port=38019
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:36261
--ranger_config_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.166.148:58929/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.166.148:58929/kms with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:04.258690 13636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:04.259032 13636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:04.259146 13636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:04.263255 13636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:42:04.263365 13636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:04.263401 13636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:04.263440 13636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:42:04.263475 13636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:42:04.269322 13636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.166.148:58929/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:36261
--ranger_config_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:36261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=38019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:04.270480 13636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:04.271564 13636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:04.280725 13642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:04.280763 13636 server_base.cc:1034] running on GCE node
W20260806 04:42:04.280716 13644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:04.280717 13641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:04.281610 13636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:04.282797 13636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:04.283998 13636 hybrid_clock.cc:648] HybridClock initialized: now 1785991324283964 us; error 51 us; skew 500 ppm
Aug 06 04:42:04 dist-test-slave-vkjp krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991324, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:04.286949 13636 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:42:04.287886 13636 webserver.cc:460] Webserver started at http://127.31.166.190:38019/ using document root <none> and password file <none>
I20260806 04:42:04.288457 13636 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:04.288525 13636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 04:42:04 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991324, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:42:04.333007 13636 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:42:04.335670 13652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:04.337244 13636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:42:04.337365 13636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "a21945364f0e48c2a4a226ad95cee176"
format_stamp: "Formatted at 2026-08-06 04:42:02 on dist-test-slave-vkjp"
server_key: "9e2bda51fc49263442a3e41cc0ca6af6"
server_key_iv: "68895ecd91034bf73691b333d22b6390"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:04.337954 13636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:04.362524 13636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:04.365386 13636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:04.365592 13636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:04.373016 13636 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:36261
I20260806 04:42:04.373027 13707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:36261 every 8 connection(s)
I20260806 04:42:04.374110 13636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:42:04.376467 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 13636
I20260806 04:42:04.377004 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.166.148:58929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:35417
--local_ip_for_outbound_sockets=127.31.166.129
--tserver_master_addrs=127.31.166.190:36261
--webserver_port=39779
--webserver_interface=127.31.166.129
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.166.148:58929/kms with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:42:04.379141 13708 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:42:04.380822 13708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176: Bootstrap starting.
I20260806 04:42:04.397426 13708 log.cc:824] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:04.404232 13708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:42:04.404660 13708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176: Bootstrap complete.
I20260806 04:42:04.408849 13708 raft_consensus.cc:348] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21945364f0e48c2a4a226ad95cee176" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36261 } }
I20260806 04:42:04.409241 13708 raft_consensus.cc:729] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a21945364f0e48c2a4a226ad95cee176, State: Initialized, Role: FOLLOWER
I20260806 04:42:04.409739 13708 consensus_queue.cc:260] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21945364f0e48c2a4a226ad95cee176" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36261 } }
I20260806 04:42:04.409863 13708 raft_consensus.cc:388] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:42:04.409951 13708 raft_consensus.cc:482] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:42:04.410045 13708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:42:04.412129 13708 raft_consensus.cc:504] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21945364f0e48c2a4a226ad95cee176" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36261 } }
I20260806 04:42:04.412470 13708 leader_election.cc:302] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a21945364f0e48c2a4a226ad95cee176; no voters:
I20260806 04:42:04.412828 13708 leader_election.cc:288] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 04:42:04.412914 13713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:42:04.413164 13713 raft_consensus.cc:686] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [term 2 LEADER]: Becoming Leader. State: Replica: a21945364f0e48c2a4a226ad95cee176, State: Running, Role: LEADER
I20260806 04:42:04.413545 13713 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21945364f0e48c2a4a226ad95cee176" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36261 } }
I20260806 04:42:04.414108 13708 sys_catalog.cc:564] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:42:04.415119 13715 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21945364f0e48c2a4a226ad95cee176. Latest consensus state: current_term: 2 leader_uuid: "a21945364f0e48c2a4a226ad95cee176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21945364f0e48c2a4a226ad95cee176" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36261 } } }
I20260806 04:42:04.415251 13715 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:04.415366 13714 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a21945364f0e48c2a4a226ad95cee176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21945364f0e48c2a4a226ad95cee176" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36261 } } }
I20260806 04:42:04.415474 13714 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176 [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:04.415920 13720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:42:04.419482 13720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:42:04.420014 13708 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 04:42:04.420163 13720 catalog_manager.cc:1259] Loaded cluster ID: c8298f05cf72442eb3c052a596076e60
I20260806 04:42:04.420281 13720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:42:04.426043 13720 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:42:04.427754 13720 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176: Loaded TSK: 0
I20260806 04:42:04.429337 13720 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:42:04.515223 13711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:04.515743 13711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:04.515892 13711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:04.522729 13711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:04.522876 13711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:04.523023 13711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:42:04.531544 13711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.166.148:58929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:35417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=39779
--tserver_master_addrs=127.31.166.190:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:04.532918 13711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:04.534320 13711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:04.546769 13749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:04.546913 13751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:04.546792 13748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:04.548202 13711 server_base.cc:1034] running on GCE node
I20260806 04:42:04.548666 13711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:04.549469 13711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:04.550645 13711 hybrid_clock.cc:648] HybridClock initialized: now 1785991324550617 us; error 45 us; skew 500 ppm
Aug 06 04:42:04 dist-test-slave-vkjp krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991324, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:04.553593 13711 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:42:04.554616 13711 webserver.cc:460] Webserver started at http://127.31.166.129:39779/ using document root <none> and password file <none>
I20260806 04:42:04.555354 13711 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:04.555522 13711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 04:42:04 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991324, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:42:04.582078 13711 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:42:04.586411 13759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:04.588195 13711 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260806 04:42:04.588342 13711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "40ffb574abdd448a9cfdc4cb5e52d934"
format_stamp: "Formatted at 2026-08-06 04:42:03 on dist-test-slave-vkjp"
server_key: "b2362f351099245ee5b007e656fb8541"
server_key_iv: "61737e419491e800c70b69c8a58a230b"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:04.589128 13711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:04.610383 13711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:04.614099 13711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:04.614337 13711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:04.615036 13711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:04.616190 13711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:04.616258 13711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:04.616328 13711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:04.616362 13711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:04.627967 13711 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:35417
I20260806 04:42:04.628865 13877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:35417 every 8 connection(s)
I20260806 04:42:04.629289 13711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:42:04.635831 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 13711
I20260806 04:42:04.636435 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.166.148:58929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:38717
--local_ip_for_outbound_sockets=127.31.166.130
--tserver_master_addrs=127.31.166.190:36261
--webserver_port=43565
--webserver_interface=127.31.166.130
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.166.148:58929/kms with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:42:04 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991324, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:04.651670 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:04.631618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:51421 (local address 127.31.166.190:36261)
0806 04:42:04.632254 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:04.632277 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:04.633629 (+ 1352us) server_negotiation.cc:407] Connection header received
0806 04:42:04.633924 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:04.633935 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:04.634247 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:04.634560 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:04.636384 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:04.638219 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:04.640200 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:04.641027 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:04.641194 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:04.644766 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:04.644774 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:04.644793 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:04.644836 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:04.647134 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:04.647892 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:04.647902 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:04.647911 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:04.648079 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:04.648581 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:04.648587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:04.648592 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:04.649257 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:04.649410 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:04.650232 (+ 822us) server_negotiation.cc:299] Negotiation successful
0806 04:42:04.650573 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":133,"threads_started":1}
I20260806 04:42:04.653396 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:04.631789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36261 (local address 127.31.166.129:51421)
0806 04:42:04.632532 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:04.632585 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:04.633388 (+ 803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:04.634984 (+ 1596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:04.634990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:04.635380 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:04.636173 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:04.636197 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:04.638376 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:04.638382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:04.640043 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:04.640060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:04.641327 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:04.641332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:04.641530 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:04.642616 (+ 1086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:04.642642 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:04.644494 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:04.647289 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:04.647299 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:04.647318 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:04.647723 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:04.648218 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:04.648224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:04.648228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:04.648446 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:04.649489 (+ 1043us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:04.649496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:04.649852 (+ 356us) client_negotiation.cc:769] Sending connection context
0806 04:42:04.652225 (+ 2373us) client_negotiation.cc:240] Negotiation successful
0806 04:42:04.652548 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":132,"threads_started":1}
I20260806 04:42:04.656751 13878 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36261
I20260806 04:42:04.657037 13878 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:04.657558 13878 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
W20260806 04:42:04.659678 13878 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:36261 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 04:42:04.659825 13878 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:04.660022 13878 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
I20260806 04:42:04.660930 13878 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:04.661131 13878 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
W20260806 04:42:04.661916 13878 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:42:04.785216 13882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:04.785514 13882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:04.785600 13882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:04.790004 13882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:04.790093 13882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:04.790194 13882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:42:04.796217 13882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.166.148:58929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:38717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=43565
--tserver_master_addrs=127.31.166.190:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:04.797374 13882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:04.798465 13882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:04.810433 13889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:04.810433 13888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:04.810487 13891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:04.810834 13882 server_base.cc:1034] running on GCE node
I20260806 04:42:04.811335 13882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:04.812141 13882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:04.813350 13882 hybrid_clock.cc:648] HybridClock initialized: now 1785991324813325 us; error 41 us; skew 500 ppm
Aug 06 04:42:04 dist-test-slave-vkjp krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991324, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:04.816818 13882 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:42:04.817968 13882 webserver.cc:460] Webserver started at http://127.31.166.130:43565/ using document root <none> and password file <none>
I20260806 04:42:04.818670 13882 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:04.818778 13882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 04:42:04 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991324, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:42:04.842224 13882 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 04:42:04.845752 13899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:04.847607 13882 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:42:04.847770 13882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "ecf753b9ee334eec80d5aaaea61f4e8e"
format_stamp: "Formatted at 2026-08-06 04:42:03 on dist-test-slave-vkjp"
server_key: "e82d3e5042f7d9c4c8778e7aabeef11c"
server_key_iv: "49d022ac4ab6ed9dd2283eab56b4860a"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:04.848593 13882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:04.877118 13882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:04.881343 13882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:04.881556 13882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:04.882305 13882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:04.883500 13882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:04.883574 13882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:04.883632 13882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:04.883664 13882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:04.895356 13882 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:38717
I20260806 04:42:04.895371 14016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:38717 every 8 connection(s)
I20260806 04:42:04.897061 13882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:42:04.907037 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 13882
I20260806 04:42:04.907580 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.166.148:58929/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:38491
--local_ip_for_outbound_sockets=127.31.166.131
--tserver_master_addrs=127.31.166.190:36261
--webserver_port=41671
--webserver_interface=127.31.166.131
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:36837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.166.148:58929/kms with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:42:04 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991324, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:04.922998 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:04.906785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:56275 (local address 127.31.166.190:36261)
0806 04:42:04.906949 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:04.906954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:04.909978 (+ 3024us) server_negotiation.cc:407] Connection header received
0806 04:42:04.911614 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:04.911617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:04.911676 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:04.911804 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:04.913092 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:04.913656 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:04.915331 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:04.915668 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:04.915792 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:04.918521 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:04.918525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:04.918528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:04.918563 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:04.920343 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:04.921014 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:04.921018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:04.921020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:04.921114 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:04.921580 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:04.921583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:04.921585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:04.921828 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:04.921910 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:04.922669 (+ 759us) server_negotiation.cc:299] Negotiation successful
0806 04:42:04.922815 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:42:04.924767 14019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:04.905600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36261 (local address 127.31.166.130:56275)
0806 04:42:04.909723 (+ 4123us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:04.909803 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:04.911339 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:04.912096 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:04.912104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:04.912397 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:04.912917 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:04.912931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:04.913812 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:04.913817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:04.915202 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:04.915213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:04.915878 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:04.915885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:04.916073 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:04.916875 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:04.916891 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:04.918345 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:04.920499 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:04.920506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:04.920522 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:04.920872 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:04.921242 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:04.921245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:04.921247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:04.921453 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:04.922000 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:04.922007 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:04.922360 (+ 353us) client_negotiation.cc:769] Sending connection context
0806 04:42:04.923302 (+ 942us) client_negotiation.cc:240] Negotiation successful
0806 04:42:04.923667 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"thread_start_us":151,"threads_started":1}
I20260806 04:42:04.926116 14017 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36261
I20260806 04:42:04.926412 14017 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:04.927023 14017 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
W20260806 04:42:04.928747 14017 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:36261 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 04:42:04.928885 14017 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:04.929071 14017 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
I20260806 04:42:04.929967 14017 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:04.930132 14017 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
W20260806 04:42:04.930907 14017 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:42:05.053177 14020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:05.053503 14020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:05.053579 14020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:05.059710 14020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:05.059813 14020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:05.059909 14020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:42:05.067270 14020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.166.148:58929/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:38491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=41671
--tserver_master_addrs=127.31.166.190:36261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:05.068704 14020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:05.069747 14020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:05.079949 14029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:05.080425 14031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:05.080678 14028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:05.080953 14020 server_base.cc:1034] running on GCE node
I20260806 04:42:05.081532 14020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:05.082472 14020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:05.083698 14020 hybrid_clock.cc:648] HybridClock initialized: now 1785991325083676 us; error 37 us; skew 500 ppm
Aug 06 04:42:05 dist-test-slave-vkjp krb5kdc[12614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991325, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:05.087730 14020 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:42:05.089133 14020 webserver.cc:460] Webserver started at http://127.31.166.131:41671/ using document root <none> and password file <none>
I20260806 04:42:05.090029 14020 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:05.090150 14020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 04:42:05 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991325, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:42:05.119168 14020 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.002s sys 0.002s
I20260806 04:42:05.122788 14039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:05.124552 14020 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260806 04:42:05.124730 14020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "075953f9b3a348f08ed2701343a5a136"
format_stamp: "Formatted at 2026-08-06 04:42:03 on dist-test-slave-vkjp"
server_key: "b0cb8741568eec352e1d30ad622c2c16"
server_key_iv: "92f06f944c420f2c51b9d0562a0d75ab"
server_key_version: "kuduclusterkey@0"
I20260806 04:42:05.125560 14020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:05.169134 14020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:05.173054 14020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:05.173290 14020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:05.173974 14020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:05.174993 14020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:05.175061 14020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:05.175155 14020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:05.175190 14020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:05.187819 14020 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:38491
I20260806 04:42:05.189306 14020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:42:05.189987 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 14020
I20260806 04:42:05.194660 14160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:38491 every 8 connection(s)
Aug 06 04:42:05 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991325, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:05.208007 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:05.193260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:42623 (local address 127.31.166.190:36261)
0806 04:42:05.193387 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:05.193392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:05.193411 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:42:05.194173 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:05.194176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:05.194237 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:05.194314 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:05.195821 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:05.196635 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:05.197916 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:05.198252 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:05.198335 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:05.201932 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:05.201936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:05.201939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:05.201967 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:05.203748 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:05.205675 (+ 1927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:05.205679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:05.205681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:05.205809 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:05.206337 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:05.206340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:05.206342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:05.206608 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:05.206685 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:05.207695 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0806 04:42:05.207834 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:42:05.211138 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:05.192241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36261 (local address 127.31.166.131:42623)
0806 04:42:05.192797 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:05.192851 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:05.193928 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:05.194496 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:05.194503 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:05.194875 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:05.195625 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:05.195659 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:05.196782 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:05.196785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:05.197746 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:05.197756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:05.198552 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:05.198555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:05.198764 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:05.199834 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:05.199850 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:05.201727 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:05.204211 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:05.204218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:05.204235 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:05.204604 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:05.205909 (+ 1305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:05.205912 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:05.205914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:05.206107 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:05.206931 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:05.206938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:05.207313 (+ 375us) client_negotiation.cc:769] Sending connection context
0806 04:42:05.209621 (+ 2308us) client_negotiation.cc:240] Negotiation successful
0806 04:42:05.210009 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":151,"threads_started":1}
I20260806 04:42:05.212553 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:05.194446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.31.166.190:36261)
0806 04:42:05.194741 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:05.194746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:05.194876 (+ 130us) server_negotiation.cc:407] Connection header received
0806 04:42:05.194926 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:05.194929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:05.194968 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:05.196069 (+ 1101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:05.196154 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:05.196853 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:05.199189 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:05.199563 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:05.199673 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:05.202308 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:05.202312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:05.202315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:05.202362 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:05.206217 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:05.206721 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:05.206723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:05.206725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:05.206795 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:05.210390 (+ 3595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:05.210394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:05.210395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:05.210648 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:05.210875 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:05.212214 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0806 04:42:05.212386 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":110,"threads_started":1}
I20260806 04:42:05.212630 14161 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36261
I20260806 04:42:05.212889 14161 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:05.213490 14161 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
W20260806 04:42:05.215178 14161 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:36261 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 04:42:05.215342 14161 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:05.215538 14161 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
I20260806 04:42:05.216406 14161 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:05.216590 14161 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
W20260806 04:42:05.217430 14161 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 04:42:05 dist-test-slave-vkjp krb5kdc[12614](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785991325, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:05.662197 13878 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:05.662644 13878 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
I20260806 04:42:05.664108 13670 ts_manager.cc:194] Registered new tserver with Master: 40ffb574abdd448a9cfdc4cb5e52d934 (127.31.166.129:35417)
I20260806 04:42:05.665825 13670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:51421
I20260806 04:42:05.931229 14017 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:05.931699 14017 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
I20260806 04:42:05.932708 13670 ts_manager.cc:194] Registered new tserver with Master: ecf753b9ee334eec80d5aaaea61f4e8e (127.31.166.130:38717)
I20260806 04:42:05.933387 13670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:56275
Aug 06 04:42:06 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991325, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:42:06.217703 14161 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:06.218094 14161 heartbeater.cc:511] Master 127.31.166.190:36261 requested a full tablet report, sending...
I20260806 04:42:06.218890 13670 ts_manager.cc:194] Registered new tserver with Master: 075953f9b3a348f08ed2701343a5a136 (127.31.166.131:38491)
I20260806 04:42:06.219625 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:42:06.220140 13670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:42623
I20260806 04:42:06.232160 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:06.221379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50172 (local address 127.31.166.190:36261)
0806 04:42:06.222033 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:06.222037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:06.222390 (+ 353us) server_negotiation.cc:407] Connection header received
0806 04:42:06.222434 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:06.222437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:06.222487 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:06.222565 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:06.223559 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.224411 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.225611 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.225968 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.226068 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:06.227894 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:06.227898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:06.227901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:06.227931 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:06.229881 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:06.230415 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:06.230419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:06.230421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:06.230531 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:06.230923 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:06.230927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:06.230929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:06.231279 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:06.231438 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:06.231711 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 04:42:06.231920 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":77,"threads_started":1}
I20260806 04:42:06.236788 13670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:42:06.241627 13670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:42:06.264127 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:06.254770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:35417 (local address 127.0.0.1:55046)
0806 04:42:06.257715 (+ 2945us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:06.257784 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:06.257929 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:06.258763 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:06.258782 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:06.258808 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:06.258948 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:06.258955 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.260479 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.260483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:06.261917 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:06.261928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.263502 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.263506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:06.263626 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:06.263667 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 04:42:06.263795 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 04:42:06.263917 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2758,"thread_start_us":108,"threads_started":1}
I20260806 04:42:06.264576 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:06.255353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55046 (local address 127.31.166.129:35417)
0806 04:42:06.255704 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:06.255715 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:06.258086 (+ 2371us) server_negotiation.cc:407] Connection header received
0806 04:42:06.258172 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:06.258177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:06.258410 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:06.258566 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:06.259119 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.260315 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.262081 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.263005 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.263279 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:06.263441 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:06.264328 (+ 887us) server_negotiation.cc:299] Negotiation successful
0806 04:42:06.264436 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":112,"threads_started":1}
I20260806 04:42:06.266147 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:06.256956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:38491 (local address 127.0.0.1:41782)
0806 04:42:06.259119 (+ 2163us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:06.259132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:06.259216 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:06.259965 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:06.259968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:06.259984 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:06.260066 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:06.260072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.262094 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.262097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:06.263260 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:06.263270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.265786 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.265790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:06.265877 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:06.265899 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:42:06.265963 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 04:42:06.266026 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2078,"thread_start_us":97,"threads_started":1}
I20260806 04:42:06.266812 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:06.257484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.31.166.131:38491)
0806 04:42:06.258220 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:06.258227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:06.259443 (+ 1216us) server_negotiation.cc:407] Connection header received
0806 04:42:06.259521 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:06.259526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:06.259697 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:06.259838 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:06.260180 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.261961 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.263408 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.265480 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.265590 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:06.265717 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:06.266579 (+ 862us) server_negotiation.cc:299] Negotiation successful
0806 04:42:06.266676 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":116,"threads_started":1}
I20260806 04:42:06.267880 13807 tablet_service.cc:1467] Processing CreateTablet for tablet ca4b11000a32438db73fd5e545234118 (DEFAULT_TABLE table=test-table [id=410196575a794857bc9f43848a03063e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:06.269078 13807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4b11000a32438db73fd5e545234118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:06.269611 14071 tablet_service.cc:1467] Processing CreateTablet for tablet ca4b11000a32438db73fd5e545234118 (DEFAULT_TABLE table=test-table [id=410196575a794857bc9f43848a03063e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:06.270604 14071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4b11000a32438db73fd5e545234118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:06.275740 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:06.257232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:38717 (local address 127.0.0.1:37148)
0806 04:42:06.263292 (+ 6060us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:06.263307 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:06.263402 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:06.271174 (+ 7772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:06.271178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:06.271201 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:06.271323 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:06.271330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.272794 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.272798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:06.273963 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:06.273976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.275248 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.275251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:06.275359 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:06.275421 (+ 62us) client_negotiation.cc:769] Sending connection context
0806 04:42:06.275508 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 04:42:06.275585 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5987,"thread_start_us":97,"threads_started":1}
I20260806 04:42:06.276271 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:06.257838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.31.166.130:38717)
0806 04:42:06.259396 (+ 1558us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:06.259403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:06.263498 (+ 4095us) server_negotiation.cc:407] Connection header received
0806 04:42:06.263588 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:06.263593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:06.263784 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:06.263920 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:06.271486 (+ 7566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.272650 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.274128 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.274895 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.275021 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:06.275177 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:06.276022 (+ 845us) server_negotiation.cc:299] Negotiation successful
0806 04:42:06.276122 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1468,"thread_start_us":89,"threads_started":1}
Aug 06 04:42:06 dist-test-slave-vkjp krb5kdc[12614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991325, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for HTTP/127.31.166.148@KRBTEST.COM
I20260806 04:42:06.279790 13944 tablet_service.cc:1467] Processing CreateTablet for tablet ca4b11000a32438db73fd5e545234118 (DEFAULT_TABLE table=test-table [id=410196575a794857bc9f43848a03063e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:06.280891 13944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca4b11000a32438db73fd5e545234118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:06.296465 14195 tablet_bootstrap.cc:492] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934: Bootstrap starting.
I20260806 04:42:06.299435 14195 tablet_bootstrap.cc:654] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:06.300348 14195 log.cc:824] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:06.303041 14197 tablet_bootstrap.cc:492] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e: Bootstrap starting.
I20260806 04:42:06.304677 14195 tablet_bootstrap.cc:492] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934: No bootstrap required, opened a new log
I20260806 04:42:06.304992 14195 ts_tablet_manager.cc:1397] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 04:42:06.305646 14197 tablet_bootstrap.cc:654] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:06.306536 14197 log.cc:824] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:06.307365 14196 tablet_bootstrap.cc:492] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136: Bootstrap starting.
I20260806 04:42:06.309460 14195 raft_consensus.cc:348] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 35417 } } peers { permanent_uuid: "075953f9b3a348f08ed2701343a5a136" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38491 } } peers { permanent_uuid: "ecf753b9ee334eec80d5aaaea61f4e8e" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38717 } }
I20260806 04:42:06.309798 14196 tablet_bootstrap.cc:654] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:06.309908 14195 raft_consensus.cc:374] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:06.310016 14195 raft_consensus.cc:729] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ffb574abdd448a9cfdc4cb5e52d934, State: Initialized, Role: FOLLOWER
I20260806 04:42:06.310601 14195 consensus_queue.cc:260] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 35417 } } peers { permanent_uuid: "075953f9b3a348f08ed2701343a5a136" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38491 } } peers { permanent_uuid: "ecf753b9ee334eec80d5aaaea61f4e8e" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38717 } }
I20260806 04:42:06.310671 14196 log.cc:824] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:06.311884 14195 ts_tablet_manager.cc:1428] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:42:06.312326 13878 heartbeater.cc:503] Master 127.31.166.190:36261 was elected leader, sending a full tablet report...
I20260806 04:42:06.314570 14197 tablet_bootstrap.cc:492] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e: No bootstrap required, opened a new log
I20260806 04:42:06.314783 14197 ts_tablet_manager.cc:1397] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260806 04:42:06.318959 14197 raft_consensus.cc:348] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 35417 } } peers { permanent_uuid: "075953f9b3a348f08ed2701343a5a136" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38491 } } peers { permanent_uuid: "ecf753b9ee334eec80d5aaaea61f4e8e" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38717 } }
I20260806 04:42:06.319358 14197 raft_consensus.cc:374] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:06.319465 14197 raft_consensus.cc:729] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf753b9ee334eec80d5aaaea61f4e8e, State: Initialized, Role: FOLLOWER
I20260806 04:42:06.320031 14197 consensus_queue.cc:260] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 35417 } } peers { permanent_uuid: "075953f9b3a348f08ed2701343a5a136" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38491 } } peers { permanent_uuid: "ecf753b9ee334eec80d5aaaea61f4e8e" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38717 } }
I20260806 04:42:06.320633 14196 tablet_bootstrap.cc:492] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136: No bootstrap required, opened a new log
I20260806 04:42:06.320904 14196 ts_tablet_manager.cc:1397] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20260806 04:42:06.321488 14197 ts_tablet_manager.cc:1428] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 04:42:06.321921 14017 heartbeater.cc:503] Master 127.31.166.190:36261 was elected leader, sending a full tablet report...
I20260806 04:42:06.324954 14196 raft_consensus.cc:348] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 35417 } } peers { permanent_uuid: "075953f9b3a348f08ed2701343a5a136" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38491 } } peers { permanent_uuid: "ecf753b9ee334eec80d5aaaea61f4e8e" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38717 } }
I20260806 04:42:06.325299 14196 raft_consensus.cc:374] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:06.325398 14196 raft_consensus.cc:729] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075953f9b3a348f08ed2701343a5a136, State: Initialized, Role: FOLLOWER
I20260806 04:42:06.325955 14196 consensus_queue.cc:260] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 35417 } } peers { permanent_uuid: "075953f9b3a348f08ed2701343a5a136" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38491 } } peers { permanent_uuid: "ecf753b9ee334eec80d5aaaea61f4e8e" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38717 } }
I20260806 04:42:06.327199 14196 ts_tablet_manager.cc:1428] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:42:06.327641 14161 heartbeater.cc:503] Master 127.31.166.190:36261 was elected leader, sending a full tablet report...
W20260806 04:42:06.382736 13879 tablet.cc:2369] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:42:06.405990 14018 tablet.cc:2369] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:42:06.446201 14162 tablet.cc:2369] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:42:06.458696 14200 raft_consensus.cc:482] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:42:06.458940 14200 raft_consensus.cc:504] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 35417 } } peers { permanent_uuid: "075953f9b3a348f08ed2701343a5a136" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38491 } } peers { permanent_uuid: "ecf753b9ee334eec80d5aaaea61f4e8e" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38717 } }
I20260806 04:42:06.460494 14200 leader_election.cc:288] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075953f9b3a348f08ed2701343a5a136 (127.31.166.131:38491), ecf753b9ee334eec80d5aaaea61f4e8e (127.31.166.130:38717)
I20260806 04:42:06.465986 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:06.461214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:42251 (local address 127.31.166.131:38491)
0806 04:42:06.461361 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:06.461366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:06.461522 (+ 156us) server_negotiation.cc:407] Connection header received
0806 04:42:06.461743 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:06.461746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:06.461794 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:06.461841 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:06.462269 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.463061 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.464697 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.465291 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.465386 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:06.465439 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:06.465769 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 04:42:06.465849 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:42:06.466658 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:06.460740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:38491 (local address 127.31.166.129:42251)
0806 04:42:06.461429 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:06.461444 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:06.461613 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:06.461988 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:06.461991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:06.462007 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:06.462149 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:06.462155 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.463234 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.463238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:06.464554 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:06.464565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.465533 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.465536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:06.465638 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:06.465660 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:42:06.466435 (+ 775us) client_negotiation.cc:240] Negotiation successful
0806 04:42:06.466517 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":137,"threads_started":1}
I20260806 04:42:06.467548 14092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4b11000a32438db73fd5e545234118" candidate_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075953f9b3a348f08ed2701343a5a136" is_pre_election: true
I20260806 04:42:06.467880 14092 raft_consensus.cc:2454] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40ffb574abdd448a9cfdc4cb5e52d934 in term 0.
I20260806 04:42:06.468576 13762 leader_election.cc:302] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075953f9b3a348f08ed2701343a5a136, 40ffb574abdd448a9cfdc4cb5e52d934; no voters:
I20260806 04:42:06.468920 14200 raft_consensus.cc:2781] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:42:06.469004 14200 raft_consensus.cc:482] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:42:06.469050 14200 raft_consensus.cc:3037] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:06.470458 14200 raft_consensus.cc:504] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 35417 } } peers { permanent_uuid: "075953f9b3a348f08ed2701343a5a136" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38491 } } peers { permanent_uuid: "ecf753b9ee334eec80d5aaaea61f4e8e" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38717 } }
I20260806 04:42:06.470839 14200 leader_election.cc:288] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [CANDIDATE]: Term 1 election: Requested vote from peers 075953f9b3a348f08ed2701343a5a136 (127.31.166.131:38491), ecf753b9ee334eec80d5aaaea61f4e8e (127.31.166.130:38717)
I20260806 04:42:06.471561 14092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4b11000a32438db73fd5e545234118" candidate_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075953f9b3a348f08ed2701343a5a136"
I20260806 04:42:06.471689 14092 raft_consensus.cc:3037] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:06.473030 14092 raft_consensus.cc:2454] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40ffb574abdd448a9cfdc4cb5e52d934 in term 1.
I20260806 04:42:06.473453 13762 leader_election.cc:302] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075953f9b3a348f08ed2701343a5a136, 40ffb574abdd448a9cfdc4cb5e52d934; no voters:
I20260806 04:42:06.473681 14200 raft_consensus.cc:2781] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:42:06.473927 14200 raft_consensus.cc:686] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [term 1 LEADER]: Becoming Leader. State: Replica: 40ffb574abdd448a9cfdc4cb5e52d934, State: Running, Role: LEADER
I20260806 04:42:06.474334 14200 consensus_queue.cc:237] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 35417 } } peers { permanent_uuid: "075953f9b3a348f08ed2701343a5a136" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38491 } } peers { permanent_uuid: "ecf753b9ee334eec80d5aaaea61f4e8e" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38717 } }
I20260806 04:42:06.478098 13670 catalog_manager.cc:5495] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40ffb574abdd448a9cfdc4cb5e52d934 (127.31.166.129). New cstate: current_term: 1 leader_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 35417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "075953f9b3a348f08ed2701343a5a136" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecf753b9ee334eec80d5aaaea61f4e8e" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38717 } health_report { overall_health: UNKNOWN } } }
I20260806 04:42:06.480767 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:06.461498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:34123 (local address 127.31.166.130:38717)
0806 04:42:06.461626 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:06.461631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:06.468552 (+ 6921us) server_negotiation.cc:407] Connection header received
0806 04:42:06.468683 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:06.468688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:06.468743 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:06.468836 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:06.469248 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.469995 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.479257 (+ 9262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.479927 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.480040 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:06.480094 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:06.480533 (+ 439us) server_negotiation.cc:299] Negotiation successful
0806 04:42:06.480641 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:42:06.481693 14207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:06.461055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:38717 (local address 127.31.166.129:34123)
0806 04:42:06.468438 (+ 7383us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:06.468456 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:06.468584 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:06.468957 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:06.468960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:06.468977 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:06.469101 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:06.469108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.470137 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.470140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:06.471299 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:06.471309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.480205 (+ 8896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.480208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:06.480339 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:06.480372 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 04:42:06.481431 (+ 1059us) client_negotiation.cc:240] Negotiation successful
0806 04:42:06.481533 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7302,"thread_start_us":110,"threads_started":1}
I20260806 04:42:06.482851 13969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4b11000a32438db73fd5e545234118" candidate_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf753b9ee334eec80d5aaaea61f4e8e" is_pre_election: true
I20260806 04:42:06.483211 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca4b11000a32438db73fd5e545234118" candidate_uuid: "40ffb574abdd448a9cfdc4cb5e52d934" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecf753b9ee334eec80d5aaaea61f4e8e"
I20260806 04:42:06.484252 13969 raft_consensus.cc:2454] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40ffb574abdd448a9cfdc4cb5e52d934 in term 0.
I20260806 04:42:06.510807 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:06.505664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.31.166.129:35417)
0806 04:42:06.505793 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:06.505797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:06.505812 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:42:06.505901 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:06.505903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:06.505958 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:06.506092 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:42:06.506394 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.507190 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.508475 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:06.508814 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:06.508910 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:06.509343 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:42:06.509983 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:42:06.510111 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:06.510569 (+ 458us) server_negotiation.cc:299] Negotiation successful
0806 04:42:06.510635 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:42:06.518685 13969 raft_consensus.cc:3037] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:06.520409 13969 raft_consensus.cc:1261] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e [term 1 FOLLOWER]: Refusing update from remote peer 40ffb574abdd448a9cfdc4cb5e52d934: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:42:06.521162 14200 consensus_queue.cc:1035] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecf753b9ee334eec80d5aaaea61f4e8e" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 38717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:42:06.522063 14092 raft_consensus.cc:1261] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136 [term 1 FOLLOWER]: Refusing update from remote peer 40ffb574abdd448a9cfdc4cb5e52d934: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:42:06.522697 14200 consensus_queue.cc:1035] T ca4b11000a32438db73fd5e545234118 P 40ffb574abdd448a9cfdc4cb5e52d934 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075953f9b3a348f08ed2701343a5a136" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 38491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:42:06.535496 14211 mvcc.cc:205] Tried to move back new op lower bound from 7315420473411108864 to 7315420473242632192. Current Snapshot: MvccSnapshot[applied={T|T < 7315420473411108864}]
I20260806 04:42:06.537457 14212 mvcc.cc:205] Tried to move back new op lower bound from 7315420473411108864 to 7315420473242632192. Current Snapshot: MvccSnapshot[applied={T|T < 7315420473411108864}]
I20260806 04:42:06.539176 14214 mvcc.cc:205] Tried to move back new op lower bound from 7315420473411108864 to 7315420473242632192. Current Snapshot: MvccSnapshot[applied={T|T < 7315420473411108864}]
I20260806 04:42:06.550334 13670 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:42:06.550522 13670 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:42:06.553531 13670 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a21945364f0e48c2a4a226ad95cee176: Sending DeleteTablet for 3 replicas of tablet ca4b11000a32438db73fd5e545234118
I20260806 04:42:06.555013 14071 tablet_service.cc:1514] Processing DeleteTablet for tablet ca4b11000a32438db73fd5e545234118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:06 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:41782
I20260806 04:42:06.555012 13807 tablet_service.cc:1514] Processing DeleteTablet for tablet ca4b11000a32438db73fd5e545234118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:06 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:55046
I20260806 04:42:06.555781 13944 tablet_service.cc:1514] Processing DeleteTablet for tablet ca4b11000a32438db73fd5e545234118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:06 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37148
I20260806 04:42:06.556826 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 13711
I20260806 04:42:06.559417 14222 tablet_replica.cc:330] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e: stopping tablet replica
I20260806 04:42:06.559844 14222 raft_consensus.cc:2229] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:42:06.560227 14222 raft_consensus.cc:2258] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:42:06.560675 14220 tablet_replica.cc:330] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136: stopping tablet replica
I20260806 04:42:06.561018 14220 raft_consensus.cc:2229] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:42:06.561313 14220 raft_consensus.cc:2258] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:42:06.563344 14220 ts_tablet_manager.cc:1905] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:42:06.563980 14222 ts_tablet_manager.cc:1905] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:42:06.566378 14220 ts_tablet_manager.cc:1918] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:42:06.566598 14220 log.cc:1196] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/ca4b11000a32438db73fd5e545234118
I20260806 04:42:06.566881 14222 ts_tablet_manager.cc:1918] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:42:06.567040 14220 ts_tablet_manager.cc:1939] T ca4b11000a32438db73fd5e545234118 P 075953f9b3a348f08ed2701343a5a136: Deleting consensus metadata
I20260806 04:42:06.568570 13655 catalog_manager.cc:4841] TS 075953f9b3a348f08ed2701343a5a136 (127.31.166.131:38491): tablet ca4b11000a32438db73fd5e545234118 (table test-table [id=410196575a794857bc9f43848a03063e]) successfully deleted
I20260806 04:42:06.568964 14222 log.cc:1196] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/ca4b11000a32438db73fd5e545234118
I20260806 04:42:06.569409 14222 ts_tablet_manager.cc:1939] T ca4b11000a32438db73fd5e545234118 P ecf753b9ee334eec80d5aaaea61f4e8e: Deleting consensus metadata
I20260806 04:42:06.570752 13657 catalog_manager.cc:4841] TS ecf753b9ee334eec80d5aaaea61f4e8e (127.31.166.130:38717): tablet ca4b11000a32438db73fd5e545234118 (table test-table [id=410196575a794857bc9f43848a03063e]) successfully deleted
W20260806 04:42:06.572690 13654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:35417 (error 108)
I20260806 04:42:06.572991 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 13882
I20260806 04:42:06.575901 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:06.575515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:35417 (local address 127.0.0.1:55060)
0806 04:42:06.575672 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:06.575790 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:35417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 04:42:06.576180 13654 catalog_manager.cc:4584] TS 40ffb574abdd448a9cfdc4cb5e52d934 (127.31.166.129:35417): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca4b11000a32438db73fd5e545234118: Network error: Client connection negotiation failed: client connection to 127.31.166.129:35417: connect: Connection refused (error 111)
I20260806 04:42:06.585225 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 14020
I20260806 04:42:06.601367 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 13636
I20260806 04:42:06.609359 32410 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 04:42:06.955315 32410 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 04:42:06.955474 32410 mini_ranger.cc:66] Stopping Ranger...
I20260806 04:42:07.301156 32410 mini_ranger.cc:68] Stopped Ranger
2026-08-06 04:42:07.301 UTC [12645] LOG: received smart shutdown request
2026-08-06 04:42:07.306 UTC [12645] LOG: background worker "logical replication launcher" (PID 12655) exited with exit code 1
2026-08-06 04:42:07.306 UTC [12650] LOG: shutting down
2026-08-06 04:42:07.510 UTC [12645] LOG: database system is shut down
2026-08-06T04:42:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51816 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:42:07 dist-test-slave-vkjp krb5kdc[14235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:42:07 dist-test-slave-vkjp krb5kdc[14235](info): set up 2 sockets
Aug 06 04:42:07 dist-test-slave-vkjp krb5kdc[14235](info): commencing operation
krb5kdc: starting...
W20260806 04:42:08.750602 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:42:08 dist-test-slave-vkjp krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:42:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:42:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:42:08.882647 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:44101
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:44101,127.31.166.189:41475,127.31.166.188:40117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:09.000394 14249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:09.000694 14249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:09.000736 14249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:09.004933 14249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:42:09.005007 14249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:09.005028 14249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:09.005053 14249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:42:09.005070 14249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:42:09.011058 14249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:44101,127.31.166.189:41475,127.31.166.188:40117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:44101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:09.012329 14249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:09.013360 14249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:09.022540 14257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:09.022544 14255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:09.022540 14254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:09.023026 14249 server_base.cc:1034] running on GCE node
I20260806 04:42:09.023703 14249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:09.024920 14249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:09.026151 14249 hybrid_clock.cc:648] HybridClock initialized: now 1785991329026142 us; error 36 us; skew 500 ppm
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991329, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:09.029340 14249 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:42:09.030249 14249 webserver.cc:460] Webserver started at http://127.31.166.190:40325/ using document root <none> and password file <none>
I20260806 04:42:09.030798 14249 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:09.030869 14249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:09.031044 14249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:42:09.033209 14249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "a420811098fd4babaee8a13367ea526c"
format_stamp: "Formatted at 2026-08-06 04:42:09 on dist-test-slave-vkjp"
server_key: "917c377bb9504a1896f6534987eb8a90"
server_key_iv: "32c7c964728e99022066d6c25cb9b7a6"
server_key_version: "encryptionkey@0"
I20260806 04:42:09.033777 14249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "a420811098fd4babaee8a13367ea526c"
format_stamp: "Formatted at 2026-08-06 04:42:09 on dist-test-slave-vkjp"
server_key: "917c377bb9504a1896f6534987eb8a90"
server_key_iv: "32c7c964728e99022066d6c25cb9b7a6"
server_key_version: "encryptionkey@0"
I20260806 04:42:09.037941 14249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:42:09.040696 14264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:09.042107 14249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:42:09.042223 14249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "a420811098fd4babaee8a13367ea526c"
format_stamp: "Formatted at 2026-08-06 04:42:09 on dist-test-slave-vkjp"
server_key: "917c377bb9504a1896f6534987eb8a90"
server_key_iv: "32c7c964728e99022066d6c25cb9b7a6"
server_key_version: "encryptionkey@0"
I20260806 04:42:09.042325 14249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:09.078836 14249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:09.082311 14249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:09.082485 14249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:09.089816 14249 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:44101
I20260806 04:42:09.090231 14318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:44101 every 8 connection(s)
I20260806 04:42:09.090873 14249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:42:09.094094 14320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:09.098481 14320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:42:09.100600 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 14249
I20260806 04:42:09.100812 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:42:09.101154 32410 external_mini_cluster.cc:1428] Setting key bb561d51937a6032bcdc7963adc1a0ba
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991329, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.166.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.189@KRBTEST.COM" created.
I20260806 04:42:09.116104 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.101148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:44101 (local address 127.0.0.1:54922)
0806 04:42:09.101791 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.101865 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:09.102661 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:09.103690 (+ 1029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:09.103696 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:09.104245 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:09.104627 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.104666 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.106506 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.106509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.107345 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.107356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.108350 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.108354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.108453 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.109094 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:09.109111 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:09.110572 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:09.112783 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.112791 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.112799 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.113056 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.113422 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.113424 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.113426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.113604 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.114274 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:09.114285 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:09.114474 (+ 189us) client_negotiation.cc:769] Sending connection context
0806 04:42:09.114689 (+ 215us) client_negotiation.cc:240] Negotiation successful
0806 04:42:09.115144 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":63,"spinlock_wait_cycles":57600,"thread_start_us":166,"threads_started":1}
I20260806 04:42:09.116111 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.101148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.31.166.190:44101)
0806 04:42:09.101717 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.101740 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.101993 (+ 253us) server_negotiation.cc:407] Connection header received
0806 04:42:09.103044 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.103051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.103409 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.103605 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.104827 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.106354 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.107489 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.108179 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.108381 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.110734 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.110744 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.110761 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.110793 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.112660 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.113168 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.113173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.113174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.113297 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.113707 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.113713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.113718 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.114141 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.114247 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.114781 (+ 534us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.115021 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":371,"thread_start_us":166,"threads_started":1}
I20260806 04:42:09.118973 14320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:42:09.119824 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.119539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:41475 (local address 127.0.0.1:54224)
0806 04:42:09.119679 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.119761 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 04:42:09.120061 14267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.166.189:41475: connect: Connection refused (error 111)
I20260806 04:42:09.120915 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.120705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:41475 (local address 127.0.0.1:54228)
0806 04:42:09.120812 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.120863 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:42:09.121098 14320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.189:41475: Network error: Client connection negotiation failed: client connection to 127.31.166.189:41475: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.189.keytab.
Entry for principal kudu/127.31.166.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.189.keytab.
I20260806 04:42:09.130539 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.189:41475
--webserver_interface=127.31.166.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.189.keytab
--principal=kudu/127.31.166.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:44101,127.31.166.189:41475,127.31.166.188:40117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:42:09.170356 14320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } attempt: 1
I20260806 04:42:09.171368 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.171025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:41475 (local address 127.0.0.1:54244)
0806 04:42:09.171216 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.171280 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 04:42:09.172140 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.171919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:41475 (local address 127.0.0.1:54252)
0806 04:42:09.172032 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.172085 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 04:42:09.172274 14320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.189:41475: Network error: Client connection negotiation failed: client connection to 127.31.166.189:41475: connect: Connection refused (error 111)
I20260806 04:42:09.240444 14320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } attempt: 2
I20260806 04:42:09.241348 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.241046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:41475 (local address 127.0.0.1:54258)
0806 04:42:09.241202 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.241269 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 04:42:09.242090 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.241876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:41475 (local address 127.0.0.1:54260)
0806 04:42:09.241983 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.242036 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 04:42:09.242221 14320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.189:41475: Network error: Client connection negotiation failed: client connection to 127.31.166.189:41475: connect: Connection refused (error 111)
W20260806 04:42:09.244441 14328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:09.244751 14328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:09.244858 14328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:09.249248 14328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:42:09.249356 14328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:09.249394 14328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:09.249452 14328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:42:09.249516 14328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:42:09.255533 14328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:44101,127.31.166.189:41475,127.31.166.188:40117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.189.keytab
--principal=kudu/127.31.166.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.189:41475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.166.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:09.256759 14328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:09.257858 14328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:09.266942 14336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:09.266929 14333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:09.266949 14334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:09.267148 14328 server_base.cc:1034] running on GCE node
I20260806 04:42:09.267884 14328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:09.269088 14328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:09.270287 14328 hybrid_clock.cc:648] HybridClock initialized: now 1785991329270263 us; error 36 us; skew 500 ppm
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991329, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:09.273427 14328 init.cc:377] Logged in from keytab as kudu/127.31.166.189@KRBTEST.COM (short username kudu)
I20260806 04:42:09.274395 14328 webserver.cc:460] Webserver started at http://127.31.166.189:34211/ using document root <none> and password file <none>
I20260806 04:42:09.274986 14328 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:09.275081 14328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:09.275359 14328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:42:09.277496 14328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/data/instance:
uuid: "d154c2a1d9664efea699f5ec89d0688c"
format_stamp: "Formatted at 2026-08-06 04:42:09 on dist-test-slave-vkjp"
server_key: "70e7a05fae4513b76e3a872c88a0a560"
server_key_iv: "7cac04bab87ec07157a49cf31c621fcb"
server_key_version: "encryptionkey@0"
I20260806 04:42:09.278085 14328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/wal/instance:
uuid: "d154c2a1d9664efea699f5ec89d0688c"
format_stamp: "Formatted at 2026-08-06 04:42:09 on dist-test-slave-vkjp"
server_key: "70e7a05fae4513b76e3a872c88a0a560"
server_key_iv: "7cac04bab87ec07157a49cf31c621fcb"
server_key_version: "encryptionkey@0"
I20260806 04:42:09.282078 14328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260806 04:42:09.284848 14343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:09.286278 14328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:42:09.286466 14328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/wal
uuid: "d154c2a1d9664efea699f5ec89d0688c"
format_stamp: "Formatted at 2026-08-06 04:42:09 on dist-test-slave-vkjp"
server_key: "70e7a05fae4513b76e3a872c88a0a560"
server_key_iv: "7cac04bab87ec07157a49cf31c621fcb"
server_key_version: "encryptionkey@0"
I20260806 04:42:09.286612 14328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:09.312042 14328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:09.315569 14328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:09.315796 14328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:09.323136 14328 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.189:41475
I20260806 04:42:09.323510 14397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.189:41475 every 8 connection(s)
I20260806 04:42:09.324156 14328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/data/info.pb
I20260806 04:42:09.327212 14398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:09.327713 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 14328
I20260806 04:42:09.327834 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-1/wal/instance
I20260806 04:42:09.328135 32410 external_mini_cluster.cc:1428] Setting key 5acd8a75846f399d4410ad06a28a8f4a
I20260806 04:42:09.331544 14398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:42:09.333398 14320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } attempt: 3
WARNING: no policy specified for kudu/127.31.166.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.188@KRBTEST.COM" created.
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991329, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for kudu/127.31.166.189@KRBTEST.COM
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991329, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.189@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:09.347429 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.333847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:41475 (local address 127.0.0.1:54270)
0806 04:42:09.334101 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.334116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:09.334223 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:09.336321 (+ 2098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:09.336326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:09.336519 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:09.336630 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.336636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.338264 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.338279 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.338900 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.338909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.339797 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.339801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.339906 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.340542 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:09.340562 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:09.342583 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:09.344692 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.344697 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.344699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.344947 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.345360 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.345363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.345365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.345455 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.346169 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:09.346175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:09.346298 (+ 123us) client_negotiation.cc:769] Sending connection context
0806 04:42:09.346436 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 04:42:09.347231 (+ 795us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":36}
I20260806 04:42:09.348333 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.334156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:44101 (local address 127.0.0.1:54934)
0806 04:42:09.334907 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.334961 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:09.335844 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:09.336401 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:09.336420 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:09.336798 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:09.337462 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.337477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.338288 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.338293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.339316 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.339336 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.340287 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.340292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.340431 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.341005 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:09.341030 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:09.342707 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:09.344982 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.344993 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.345004 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.345244 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.346415 (+ 1171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.346419 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.346421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.346501 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.346886 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:09.346892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:09.347053 (+ 161us) client_negotiation.cc:769] Sending connection context
0806 04:42:09.347233 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 04:42:09.347414 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":30,"spinlock_wait_cycles":59648,"thread_start_us":179,"threads_started":1}
I20260806 04:42:09.348342 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.334254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54270 (local address 127.31.166.189:41475)
0806 04:42:09.334690 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.334710 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.334752 (+ 42us) server_negotiation.cc:407] Connection header received
0806 04:42:09.335828 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.335851 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.336110 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.336778 (+ 668us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.337184 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.338139 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.339031 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.339663 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.339841 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.342745 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.342755 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.342781 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.342812 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.344569 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.345059 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.345068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.345070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.345232 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.345575 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.345579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.345580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.346043 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.346165 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.346812 (+ 647us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.347206 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":294,"thread_start_us":172,"threads_started":1}
I20260806 04:42:09.349177 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.333666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.31.166.190:44101)
0806 04:42:09.333910 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.333915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.335062 (+ 1147us) server_negotiation.cc:407] Connection header received
0806 04:42:09.336078 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.336082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.336160 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.336270 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.337632 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.338156 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.339453 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.339800 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.340000 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.342847 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.342854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.342860 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.342893 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.344860 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.346220 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.346225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.346227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.346317 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.346593 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.346598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.346602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.346784 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.347957 (+ 1173us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.348860 (+ 903us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.349047 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":72}
I20260806 04:42:09.350247 14398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:42:09.350723 14320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:42:09.351555 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.351185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33318)
0806 04:42:09.351427 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.351481 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260806 04:42:09.352213 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.351960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33332)
0806 04:42:09.352103 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.352155 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260806 04:42:09.352346 14320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:40117: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991329, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.189@KRBTEST.COM for kudu/127.31.166.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.188.keytab.
Entry for principal kudu/127.31.166.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.188.keytab.
I20260806 04:42:09.358940 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.188:40117
--webserver_interface=127.31.166.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.188.keytab
--principal=kudu/127.31.166.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:44459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:44101,127.31.166.189:41475,127.31.166.188:40117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:42:09.359284 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.350894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:41475 (local address 127.0.0.1:54286)
0806 04:42:09.351041 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.351058 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:09.351164 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:09.351472 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:09.351475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:09.351653 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:09.351786 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.351795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.352726 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.352728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.353410 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.353419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.353979 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.353982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.354057 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.354570 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:09.354581 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:09.355829 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:09.357535 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.357537 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.357538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.357728 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.358069 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.358074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.358076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.358154 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.358600 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:09.358605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:09.358721 (+ 116us) client_negotiation.cc:769] Sending connection context
0806 04:42:09.358845 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 04:42:09.358997 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 04:42:09.359489 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.351121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54286 (local address 127.31.166.189:41475)
0806 04:42:09.351229 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.351234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.351250 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:42:09.351290 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.351296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.351346 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.351438 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.351944 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.352628 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.353544 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.353878 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.353980 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.355952 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.355955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.355957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.355978 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.357433 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.357852 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.357856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.357858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.357959 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.358253 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.358256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.358257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.358497 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.358576 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.358906 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.359166 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":41}
I20260806 04:42:09.360126 14398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:42:09.360879 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.360594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33344)
0806 04:42:09.360756 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.360831 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260806 04:42:09.361498 14344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
I20260806 04:42:09.362397 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.362185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33348)
0806 04:42:09.362294 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.362348 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:42:09.362562 14398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:40117: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
I20260806 04:42:09.383518 14320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } attempt: 1
I20260806 04:42:09.384475 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.384154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33364)
0806 04:42:09.384321 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.384390 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 04:42:09.385268 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.385043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33366)
0806 04:42:09.385155 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.385209 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 04:42:09.385418 14320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:40117: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
I20260806 04:42:09.411820 14398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } attempt: 1
I20260806 04:42:09.412778 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.412466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33380)
0806 04:42:09.412632 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.412701 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 04:42:09.413514 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.413295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33392)
0806 04:42:09.413406 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.413461 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 04:42:09.413642 14398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:40117: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
I20260806 04:42:09.460589 14320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } attempt: 2
I20260806 04:42:09.461476 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.461177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33404)
0806 04:42:09.461331 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.461399 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 04:42:09.462198 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.461991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33412)
0806 04:42:09.462094 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.462145 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 04:42:09.462319 14320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:40117: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
W20260806 04:42:09.476563 14405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:09.476871 14405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:09.476979 14405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:09.481119 14405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:42:09.481206 14405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:09.481243 14405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:09.481305 14405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:42:09.481343 14405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:42:09.481870 14398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } attempt: 2
I20260806 04:42:09.482738 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.482430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33426)
0806 04:42:09.482575 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.482664 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 04:42:09.483525 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.483304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33442)
0806 04:42:09.483417 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.483471 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:42:09.483675 14398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:40117: Network error: Client connection negotiation failed: client connection to 127.31.166.188:40117: connect: Connection refused (error 111)
I20260806 04:42:09.487154 14405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:44459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:44101,127.31.166.189:41475,127.31.166.188:40117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.188.keytab
--principal=kudu/127.31.166.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.188:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.166.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:09.488284 14405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:09.489359 14405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:09.498350 14411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:09.498369 14414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:09.498414 14412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:09.498647 14405 server_base.cc:1034] running on GCE node
I20260806 04:42:09.499313 14405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:09.500427 14405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:09.501590 14405 hybrid_clock.cc:648] HybridClock initialized: now 1785991329501577 us; error 24 us; skew 500 ppm
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991329, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:09.504457 14405 init.cc:377] Logged in from keytab as kudu/127.31.166.188@KRBTEST.COM (short username kudu)
I20260806 04:42:09.505386 14405 webserver.cc:460] Webserver started at http://127.31.166.188:45033/ using document root <none> and password file <none>
I20260806 04:42:09.505968 14405 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:09.506060 14405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:09.506299 14405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:42:09.508404 14405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/data/instance:
uuid: "0725f792d44c4c199fe0e3e436a04a5a"
format_stamp: "Formatted at 2026-08-06 04:42:09 on dist-test-slave-vkjp"
server_key: "cafa028553744e5b1ab04d97ea47b473"
server_key_iv: "ef4f4d80a0c1a20125cc3b429f247938"
server_key_version: "encryptionkey@0"
I20260806 04:42:09.508967 14405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/wal/instance:
uuid: "0725f792d44c4c199fe0e3e436a04a5a"
format_stamp: "Formatted at 2026-08-06 04:42:09 on dist-test-slave-vkjp"
server_key: "cafa028553744e5b1ab04d97ea47b473"
server_key_iv: "ef4f4d80a0c1a20125cc3b429f247938"
server_key_version: "encryptionkey@0"
I20260806 04:42:09.512626 14405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:42:09.515054 14422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:09.516352 14405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:42:09.516500 14405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/wal
uuid: "0725f792d44c4c199fe0e3e436a04a5a"
format_stamp: "Formatted at 2026-08-06 04:42:09 on dist-test-slave-vkjp"
server_key: "cafa028553744e5b1ab04d97ea47b473"
server_key_iv: "ef4f4d80a0c1a20125cc3b429f247938"
server_key_version: "encryptionkey@0"
I20260806 04:42:09.516614 14405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:09.529194 14405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:09.532693 14405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:09.532970 14405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:09.542003 14405 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.188:40117
I20260806 04:42:09.542294 14476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.188:40117 every 8 connection(s)
I20260806 04:42:09.543239 14405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/data/info.pb
I20260806 04:42:09.546187 14477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:09.546412 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 14405
I20260806 04:42:09.546626 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785991160137184-32410-0/minicluster-data/master-2/wal/instance
I20260806 04:42:09.547015 32410 external_mini_cluster.cc:1428] Setting key e0d028af795e6471309a67bdc06d9e59
I20260806 04:42:09.550397 14477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.188@KRBTEST.COM
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991329, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.188@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:09.561416 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.552381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.31.166.190:44101)
0806 04:42:09.552536 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.552542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.552812 (+ 270us) server_negotiation.cc:407] Connection header received
0806 04:42:09.552979 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.552982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.553054 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.553138 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.553766 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.554276 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.555434 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.555667 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.555776 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.558062 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.558065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.558067 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.558091 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.559598 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.560105 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.560113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.560114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.560184 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.560451 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.560477 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.560479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.560639 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.560740 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.561113 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.561236 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:42:09.561506 14320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } attempt: 3
I20260806 04:42:09.562304 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.548546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33452 (local address 127.31.166.188:40117)
0806 04:42:09.549025 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.549043 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.549094 (+ 51us) server_negotiation.cc:407] Connection header received
0806 04:42:09.549803 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.549820 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.550238 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.550550 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.551498 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.552857 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.553848 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.554296 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.554565 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.557321 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.557328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.557341 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.557374 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.559285 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.559751 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.559755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.559756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.559921 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.560217 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.560220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.560221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.560578 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.560800 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.561003 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.561291 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":301,"thread_start_us":153,"threads_started":1}
I20260806 04:42:09.562777 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.552303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:44101 (local address 127.0.0.1:54942)
0806 04:42:09.552694 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.552732 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:09.552871 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:09.553188 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:09.553192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:09.553526 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:09.553640 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.553646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.554402 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.554405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.555308 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.555318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.555813 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.555817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.555916 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.556477 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:09.556502 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:09.557902 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:09.559717 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.559726 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.559735 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.560001 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.560279 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.560281 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.560283 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.560362 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.560753 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:09.560761 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:09.560885 (+ 124us) client_negotiation.cc:769] Sending connection context
0806 04:42:09.561055 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 04:42:09.561426 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":22,"thread_start_us":101,"threads_started":1}
I20260806 04:42:09.564248 14477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991329, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for kudu/127.31.166.188@KRBTEST.COM
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991329, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.188@KRBTEST.COM for kudu/127.31.166.189@KRBTEST.COM
I20260806 04:42:09.572381 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.562016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.31.166.188:40117)
0806 04:42:09.562352 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.562358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.562369 (+ 11us) server_negotiation.cc:407] Connection header received
0806 04:42:09.562395 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.562417 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.562458 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.562999 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.563104 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.563795 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.564829 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.565169 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.565269 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.567658 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.567663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.567665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.567689 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.570281 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.570717 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.570720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.570721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.570795 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.571146 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.571152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.571157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.571382 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.571493 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.572012 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.572137 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":155,"threads_started":1}
I20260806 04:42:09.572963 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.561920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33454)
0806 04:42:09.562034 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.562049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:09.562123 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:09.562594 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:09.562598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:09.562768 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:09.562868 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.562876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.563933 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.563936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.564706 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.564721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.565272 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.565279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.565363 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.565858 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:09.565876 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:09.567530 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:09.570386 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.570389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.570391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.570622 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.570897 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.570901 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.570903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.571008 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.571744 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:09.571747 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:09.571835 (+ 88us) client_negotiation.cc:769] Sending connection context
0806 04:42:09.571951 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 04:42:09.572773 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 04:42:09.574839 14398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } attempt: 3
I20260806 04:42:09.575404 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.565253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54294 (local address 127.31.166.189:41475)
0806 04:42:09.565383 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.565387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.565668 (+ 281us) server_negotiation.cc:407] Connection header received
0806 04:42:09.565784 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.565787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.565844 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.565919 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.566953 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.567700 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.568804 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.569242 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.569358 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.571767 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.571773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.571777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.571809 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.573647 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.574106 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.574109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.574111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.574195 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.574479 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.574484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.574486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.574689 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.574780 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.575147 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.575263 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:42:09.575404 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.564911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.31.166.190:44101)
0806 04:42:09.565231 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.565238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.565249 (+ 11us) server_negotiation.cc:407] Connection header received
0806 04:42:09.565281 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.565284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.565322 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.565396 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.566127 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.566586 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.567485 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.567786 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.567896 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.571216 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.571220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.571223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.571246 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.573056 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.573562 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.573568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.573572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.573656 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.573979 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.573983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.573984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.574184 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.574407 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.574752 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.575253 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260806 04:42:09.576107 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.565408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:41475 (local address 127.0.0.1:54294)
0806 04:42:09.565579 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.565591 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:09.565697 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:09.566046 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:09.566049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:09.566199 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:09.566778 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.566789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.567808 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.567817 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.568650 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.568663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.569365 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.569369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.569444 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.569942 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:09.569957 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:09.571560 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:09.573763 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.573766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.573768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.574001 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.574287 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.574291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.574293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.574387 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.574836 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:09.574839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:09.574929 (+ 90us) client_negotiation.cc:769] Sending connection context
0806 04:42:09.575535 (+ 606us) client_negotiation.cc:240] Negotiation successful
0806 04:42:09.575675 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 04:42:09.576696 14320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c: Bootstrap starting.
I20260806 04:42:09.579195 14477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:42:09.579732 14320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:09.580838 14320 log.cc:824] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.189@KRBTEST.COM
I20260806 04:42:09.583251 14320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c: No bootstrap required, opened a new log
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991329, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.189@KRBTEST.COM for kudu/127.31.166.188@KRBTEST.COM
I20260806 04:42:09.586239 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.576584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.31.166.189:41475)
0806 04:42:09.576873 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.576876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.576886 (+ 10us) server_negotiation.cc:407] Connection header received
0806 04:42:09.576912 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.576914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.576944 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.577002 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.577690 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.578134 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.579122 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.579492 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.579592 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.582153 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.582157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.582159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.582185 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.583974 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.584465 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.584470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.584473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.584553 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.584849 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.584852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.584854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.585048 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.585843 (+ 795us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.585926 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.586053 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
Aug 06 04:42:09 dist-test-slave-vkjp krb5kdc[14235](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991329, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.188@KRBTEST.COM for kudu/127.31.166.188@KRBTEST.COM
I20260806 04:42:09.589185 14403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.575260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33458)
0806 04:42:09.575492 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.575508 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:09.575588 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:09.579840 (+ 4252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:09.579843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:09.580022 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:09.580115 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.580123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.581177 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.581183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.581785 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.581794 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.582285 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.582289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.582373 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.583238 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:09.583252 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:09.584722 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:09.587132 (+ 2410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.587137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.587141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.587431 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.587742 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.587746 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.587747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.587836 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.588232 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:09.588235 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:09.588583 (+ 348us) client_negotiation.cc:769] Sending connection context
0806 04:42:09.588667 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 04:42:09.589027 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260806 04:42:09.589691 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.579188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.31.166.188:40117)
0806 04:42:09.579298 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.579301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.579314 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:42:09.579344 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.579347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.579382 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.580236 (+ 854us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.580317 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.581039 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.581903 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.582166 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.582273 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.584870 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.584873 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.584876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.584902 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.586949 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.587563 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.587566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.587567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.587645 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.587946 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.587948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.587950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.588129 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.589377 (+ 1248us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.589451 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.589558 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:42:09.591862 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.579847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.31.166.188:40117)
0806 04:42:09.579949 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.579955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.582515 (+ 2560us) server_negotiation.cc:407] Connection header received
0806 04:42:09.582643 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.582646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.582685 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.582755 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.583286 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.584129 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.585194 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.585530 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.585606 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.588021 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.588024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.588026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.588052 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.590235 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.590725 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.590731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.590732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.590808 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.591136 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.591139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.591141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.591308 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.591370 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.591630 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.591728 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:42:09.592464 14481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.579659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:40117 (local address 127.0.0.1:33464)
0806 04:42:09.582445 (+ 2786us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:09.582459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:09.582547 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:09.582859 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:09.582864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:09.583028 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:09.583163 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.583172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.584245 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.584250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.585084 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:09.585096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.585740 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.585743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:09.585822 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.586197 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:09.586210 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:09.587898 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:09.590357 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.590360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.590361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.590617 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.590908 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:09.590911 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:09.590913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:09.590993 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:09.591420 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:09.591423 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:09.591496 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 04:42:09.592203 (+ 707us) client_negotiation.cc:240] Negotiation successful
0806 04:42:09.592320 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2691}
I20260806 04:42:09.596362 14320 raft_consensus.cc:348] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } }
I20260806 04:42:09.596633 14320 raft_consensus.cc:374] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:09.596697 14320 raft_consensus.cc:729] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a420811098fd4babaee8a13367ea526c, State: Initialized, Role: FOLLOWER
I20260806 04:42:09.597214 14320 consensus_queue.cc:260] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } }
I20260806 04:42:09.597779 14484 sys_catalog.cc:455] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } } }
I20260806 04:42:09.597918 14484 sys_catalog.cc:458] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:42:09.598783 14320 sys_catalog.cc:564] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:42:09.600844 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:42:09.602607 14477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a: Bootstrap starting.
I20260806 04:42:09.603045 14398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c: Bootstrap starting.
I20260806 04:42:09.605458 14477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:09.606375 14477 log.cc:824] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:09.607113 14398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:09.608012 14398 log.cc:824] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:09.605860 14249 master_runner.cc:186] Error getting master registration for 127.31.166.189:41475: OK, instance_id { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" instance_seqno: 1785991329321045 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:42:09.610368 14477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a: No bootstrap required, opened a new log
I20260806 04:42:09.610499 14249 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 04:42:09.612761 14495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:42:09.612839 14495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:42:09.613173 14398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c: No bootstrap required, opened a new log
I20260806 04:42:09.617792 14398 raft_consensus.cc:348] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } }
I20260806 04:42:09.618046 14398 raft_consensus.cc:374] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:09.618116 14398 raft_consensus.cc:729] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d154c2a1d9664efea699f5ec89d0688c, State: Initialized, Role: FOLLOWER
I20260806 04:42:09.618598 14398 consensus_queue.cc:260] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } }
I20260806 04:42:09.621373 14398 sys_catalog.cc:564] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:42:09.625209 14480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.603575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.31.166.188:40117)
0806 04:42:09.603721 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.603725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.605326 (+ 1601us) server_negotiation.cc:407] Connection header received
0806 04:42:09.605524 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.605527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.605583 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.605637 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.606677 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.607466 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.608622 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.608977 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.609056 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.616753 (+ 7697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.616760 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.616764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.616795 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.621528 (+ 4733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.623768 (+ 2240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.623772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.623774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.623867 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.624203 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.624207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.624211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.624428 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.624533 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.624890 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.625030 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 04:42:09.625701 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.603213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.31.166.190:44101)
0806 04:42:09.603337 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.603341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.614287 (+ 10946us) server_negotiation.cc:407] Connection header received
0806 04:42:09.615780 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.615784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.615835 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.615890 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.617172 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.617901 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.619371 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.619670 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.619777 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.621775 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.621779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.621782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.621809 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.623597 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.624101 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.624104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.624106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.624192 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.624484 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.624488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.624490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.624720 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.624860 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.625430 (+ 570us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.625549 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 04:42:09.625947 14477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } }
I20260806 04:42:09.626233 14477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:09.626300 14477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0725f792d44c4c199fe0e3e436a04a5a, State: Initialized, Role: FOLLOWER
I20260806 04:42:09.627031 14477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } }
I20260806 04:42:09.628274 14507 sys_catalog.cc:455] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } } }
I20260806 04:42:09.628470 14507 sys_catalog.cc:458] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:42:09.628845 14477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:42:09.634176 14402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:09.604401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.31.166.189:41475)
0806 04:42:09.605001 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:09.605005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:09.614492 (+ 9487us) server_negotiation.cc:407] Connection header received
0806 04:42:09.614756 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:09.614759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:09.614807 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:09.614857 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:09.619891 (+ 5034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.620646 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.627357 (+ 6711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:09.627678 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:09.627754 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:09.629753 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:09.629759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:09.629765 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:09.629798 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:09.631786 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.632487 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.632491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.632493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.632578 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:09.632881 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:09.632887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:09.632892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:09.633123 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:09.633237 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:09.633897 (+ 660us) server_negotiation.cc:299] Negotiation successful
0806 04:42:09.634025 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504}
W20260806 04:42:09.634644 14521 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:42:09.634711 14521 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:42:09.631382 14328 master_runner.cc:186] Error getting master registration for 127.31.166.188:40117: OK, instance_id { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" instance_seqno: 1785991329539512 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:42:09.635411 14328 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:42:09.638270 14405 master_runner.cc:263] No leader master found from master 0725f792d44c4c199fe0e3e436a04a5a
I20260806 04:42:09.638342 14405 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:42:09.638504 14522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } } }
I20260806 04:42:09.638599 14522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [sys.catalog]: This master's current role is: FOLLOWER
W20260806 04:42:09.640039 14533 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:42:09.640164 14533 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:42:09.660594 14484 raft_consensus.cc:482] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:42:09.660794 14484 raft_consensus.cc:504] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } }
I20260806 04:42:09.662279 14484 leader_election.cc:288] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d154c2a1d9664efea699f5ec89d0688c (127.31.166.189:41475), 0725f792d44c4c199fe0e3e436a04a5a (127.31.166.188:40117)
I20260806 04:42:09.663319 14449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a420811098fd4babaee8a13367ea526c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0725f792d44c4c199fe0e3e436a04a5a" is_pre_election: true
I20260806 04:42:09.663654 14449 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a420811098fd4babaee8a13367ea526c in term 0.
I20260806 04:42:09.664108 14265 leader_election.cc:302] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0725f792d44c4c199fe0e3e436a04a5a, a420811098fd4babaee8a13367ea526c; no voters:
I20260806 04:42:09.664579 14370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a420811098fd4babaee8a13367ea526c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d154c2a1d9664efea699f5ec89d0688c" is_pre_election: true
I20260806 04:42:09.664875 14370 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a420811098fd4babaee8a13367ea526c in term 0.
I20260806 04:42:09.667765 14484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:42:09.667886 14484 raft_consensus.cc:482] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:42:09.667936 14484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:09.669291 14484 raft_consensus.cc:504] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } }
I20260806 04:42:09.669963 14370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a420811098fd4babaee8a13367ea526c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d154c2a1d9664efea699f5ec89d0688c"
I20260806 04:42:09.670092 14370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:09.671473 14370 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a420811098fd4babaee8a13367ea526c in term 1.
I20260806 04:42:09.671784 14267 leader_election.cc:302] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a420811098fd4babaee8a13367ea526c, d154c2a1d9664efea699f5ec89d0688c; no voters:
I20260806 04:42:09.672514 14449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a420811098fd4babaee8a13367ea526c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0725f792d44c4c199fe0e3e436a04a5a"
I20260806 04:42:09.672643 14449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:09.673936 14449 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a420811098fd4babaee8a13367ea526c in term 1.
I20260806 04:42:09.674278 14484 leader_election.cc:288] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [CANDIDATE]: Term 1 election: Requested vote from peers d154c2a1d9664efea699f5ec89d0688c (127.31.166.189:41475), 0725f792d44c4c199fe0e3e436a04a5a (127.31.166.188:40117)
I20260806 04:42:09.674431 14484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:42:09.674531 14484 raft_consensus.cc:686] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [term 1 LEADER]: Becoming Leader. State: Replica: a420811098fd4babaee8a13367ea526c, State: Running, Role: LEADER
I20260806 04:42:09.674918 14484 consensus_queue.cc:237] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } }
I20260806 04:42:09.679776 14484 sys_catalog.cc:455] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a420811098fd4babaee8a13367ea526c. Latest consensus state: current_term: 1 leader_uuid: "a420811098fd4babaee8a13367ea526c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } } }
I20260806 04:42:09.679903 14484 sys_catalog.cc:458] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:09.681670 14539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:42:09.682619 14539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:42:09.688114 14449 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [term 1 FOLLOWER]: Refusing update from remote peer a420811098fd4babaee8a13367ea526c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:42:09.688768 14484 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:42:09.692633 14522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a420811098fd4babaee8a13367ea526c. Latest consensus state: current_term: 1 leader_uuid: "a420811098fd4babaee8a13367ea526c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } } }
I20260806 04:42:09.692767 14522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:42:09.696153 14370 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [term 1 FOLLOWER]: Refusing update from remote peer a420811098fd4babaee8a13367ea526c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:42:09.696731 14537 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:42:09.701138 14507 sys_catalog.cc:455] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a420811098fd4babaee8a13367ea526c. Latest consensus state: current_term: 1 leader_uuid: "a420811098fd4babaee8a13367ea526c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } } }
I20260806 04:42:09.701272 14507 sys_catalog.cc:458] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:42:09.703899 14537 sys_catalog.cc:455] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a420811098fd4babaee8a13367ea526c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } } }
I20260806 04:42:09.704023 14537 sys_catalog.cc:458] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:09.705295 14522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a420811098fd4babaee8a13367ea526c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } } }
I20260806 04:42:09.705431 14522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:42:09.706041 14537 sys_catalog.cc:455] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a420811098fd4babaee8a13367ea526c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } } }
I20260806 04:42:09.706173 14537 sys_catalog.cc:458] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:09.707212 14507 sys_catalog.cc:455] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a420811098fd4babaee8a13367ea526c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a420811098fd4babaee8a13367ea526c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44101 } } peers { permanent_uuid: "d154c2a1d9664efea699f5ec89d0688c" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 41475 } } peers { permanent_uuid: "0725f792d44c4c199fe0e3e436a04a5a" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 40117 } } }
I20260806 04:42:09.707340 14507 sys_catalog.cc:458] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:42:09.708177 14540 mvcc.cc:205] Tried to move back new op lower bound from 7315420486394703872 to 7315420486351929344. Current Snapshot: MvccSnapshot[applied={T|T < 7315420486394703872}]
I20260806 04:42:09.708489 14543 mvcc.cc:205] Tried to move back new op lower bound from 7315420486394703872 to 7315420486351929344. Current Snapshot: MvccSnapshot[applied={T|T < 7315420486394703872}]
I20260806 04:42:09.708654 14541 mvcc.cc:205] Tried to move back new op lower bound from 7315420486394703872 to 7315420486351929344. Current Snapshot: MvccSnapshot[applied={T|T < 7315420486394703872}]
I20260806 04:42:09.712779 14539 catalog_manager.cc:1347] Generated new cluster ID: a56a9935779e429bb7a5da904a7e51cf
I20260806 04:42:09.712849 14539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:42:09.735606 14539 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:42:09.736937 14539 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:42:09.750737 14539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a420811098fd4babaee8a13367ea526c: Generated new TSK 0
I20260806 04:42:09.751576 14539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:42:10.636180 14521 catalog_manager.cc:1259] Loaded cluster ID: a56a9935779e429bb7a5da904a7e51cf
I20260806 04:42:10.636327 14521 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c: loading cluster ID for follower catalog manager: success
I20260806 04:42:10.638074 14521 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c: acquiring CA information for follower catalog manager: success
I20260806 04:42:10.639389 14521 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d154c2a1d9664efea699f5ec89d0688c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:42:10.641497 14533 catalog_manager.cc:1259] Loaded cluster ID: a56a9935779e429bb7a5da904a7e51cf
I20260806 04:42:10.641564 14533 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a: loading cluster ID for follower catalog manager: success
I20260806 04:42:10.643194 14533 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a: acquiring CA information for follower catalog manager: success
I20260806 04:42:10.644338 14533 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0725f792d44c4c199fe0e3e436a04a5a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:42:10.822052 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 14249
I20260806 04:42:10.829648 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 14328
I20260806 04:42:10.836859 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 14405
2026-08-06T04:42:10Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3201 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:42:10 dist-test-slave-vkjp krb5kdc[14556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:42:10 dist-test-slave-vkjp krb5kdc[14556](info): set up 2 sockets
Aug 06 04:42:10 dist-test-slave-vkjp krb5kdc[14556](info): commencing operation
krb5kdc: starting...
W20260806 04:42:11.919281 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:42:12 dist-test-slave-vkjp krb5kdc[14556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:42:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:42:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:42:12.050863 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:37625
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:44421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:37625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:12.163686 14569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:12.163970 14569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:12.164012 14569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:12.168167 14569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:42:12.168228 14569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:12.168247 14569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:12.168265 14569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:42:12.168280 14569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:42:12.173717 14569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:44421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:37625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:37625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:12.174816 14569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:12.175912 14569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:12.184828 14577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:12.184851 14574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:12.184931 14569 server_base.cc:1034] running on GCE node
W20260806 04:42:12.184829 14575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:12.185644 14569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:12.186858 14569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:12.188066 14569 hybrid_clock.cc:648] HybridClock initialized: now 1785991332188061 us; error 57 us; skew 500 ppm
Aug 06 04:42:12 dist-test-slave-vkjp krb5kdc[14556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991332, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:12.190897 14569 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:42:12.191907 14569 webserver.cc:460] Webserver started at http://127.31.166.190:37089/ using document root <none> and password file <none>
I20260806 04:42:12.192502 14569 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:12.192600 14569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:12.192842 14569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:42:12.195421 14569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "f83d19c64c294ff08bca6240a26e611b"
format_stamp: "Formatted at 2026-08-06 04:42:12 on dist-test-slave-vkjp"
server_key: "7ff71aa823dab1b6c8288c63811ea822"
server_key_iv: "a26823309c1bad6c27c17b79a03e25fc"
server_key_version: "encryptionkey@0"
I20260806 04:42:12.196010 14569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "f83d19c64c294ff08bca6240a26e611b"
format_stamp: "Formatted at 2026-08-06 04:42:12 on dist-test-slave-vkjp"
server_key: "7ff71aa823dab1b6c8288c63811ea822"
server_key_iv: "a26823309c1bad6c27c17b79a03e25fc"
server_key_version: "encryptionkey@0"
I20260806 04:42:12.199847 14569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:42:12.202366 14584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:12.203692 14569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:42:12.203847 14569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "f83d19c64c294ff08bca6240a26e611b"
format_stamp: "Formatted at 2026-08-06 04:42:12 on dist-test-slave-vkjp"
server_key: "7ff71aa823dab1b6c8288c63811ea822"
server_key_iv: "a26823309c1bad6c27c17b79a03e25fc"
server_key_version: "encryptionkey@0"
I20260806 04:42:12.203969 14569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:12.223044 14569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:12.226231 14569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:12.226471 14569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:12.233779 14569 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:37625
I20260806 04:42:12.233798 14641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:37625 every 8 connection(s)
I20260806 04:42:12.234745 14569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:42:12.237462 14642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:12.237906 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 14569
I20260806 04:42:12.238065 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:42:12.238368 32410 external_mini_cluster.cc:1428] Setting key 55dd308209f09b9ce202a649ab348208
I20260806 04:42:12.244632 14642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b: Bootstrap starting.
Aug 06 04:42:12 dist-test-slave-vkjp krb5kdc[14556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:12.247574 14642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:12.248440 14642 log.cc:824] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:12.250748 14642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b: No bootstrap required, opened a new log
I20260806 04:42:12.253537 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:12.239641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.31.166.190:37625)
0806 04:42:12.240068 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:12.240083 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:12.240118 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:42:12.240788 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:12.240807 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:12.241130 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:12.241412 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:12.242382 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:12.243448 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:12.244605 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:12.245110 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:12.245229 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:12.248080 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:12.248087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:12.248102 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:12.248134 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:12.250083 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:12.250597 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:12.250602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:12.250606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:12.250717 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:12.251034 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:12.251036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:12.251037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:12.251481 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:12.251631 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:12.252067 (+ 436us) server_negotiation.cc:299] Negotiation successful
0806 04:42:12.252694 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":144,"threads_started":1}
I20260806 04:42:12.255046 14642 raft_consensus.cc:348] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83d19c64c294ff08bca6240a26e611b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37625 } }
I20260806 04:42:12.255362 14642 raft_consensus.cc:374] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:12.255439 14642 raft_consensus.cc:729] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f83d19c64c294ff08bca6240a26e611b, State: Initialized, Role: FOLLOWER
I20260806 04:42:12.255923 14642 consensus_queue.cc:260] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83d19c64c294ff08bca6240a26e611b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37625 } }
I20260806 04:42:12.256088 14642 raft_consensus.cc:388] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:42:12.256181 14642 raft_consensus.cc:482] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:42:12.256286 14642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:12.256484 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:42:12.257313 14642 raft_consensus.cc:504] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83d19c64c294ff08bca6240a26e611b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37625 } }
I20260806 04:42:12.257685 14642 leader_election.cc:302] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f83d19c64c294ff08bca6240a26e611b; no voters:
I20260806 04:42:12.258042 14642 leader_election.cc:288] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:42:12.258128 14647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:42:12.258333 14647 raft_consensus.cc:686] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [term 1 LEADER]: Becoming Leader. State: Replica: f83d19c64c294ff08bca6240a26e611b, State: Running, Role: LEADER
I20260806 04:42:12.258618 14647 consensus_queue.cc:237] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83d19c64c294ff08bca6240a26e611b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37625 } }
I20260806 04:42:12.259362 14642 sys_catalog.cc:564] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:42:12.260360 14648 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f83d19c64c294ff08bca6240a26e611b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83d19c64c294ff08bca6240a26e611b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37625 } } }
I20260806 04:42:12.260507 14648 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:12.260830 14650 sys_catalog.cc:455] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f83d19c64c294ff08bca6240a26e611b. Latest consensus state: current_term: 1 leader_uuid: "f83d19c64c294ff08bca6240a26e611b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f83d19c64c294ff08bca6240a26e611b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 37625 } } }
I20260806 04:42:12.260914 14657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:42:12.260985 14650 sys_catalog.cc:458] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:12.263808 14657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:42:12.269744 14657 catalog_manager.cc:1347] Generated new cluster ID: 4d8d0990a1104dabae83d7fc384a586b
I20260806 04:42:12.269840 14657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:42:12.277488 14657 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:42:12.278505 14657 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:42:12.284577 14657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f83d19c64c294ff08bca6240a26e611b: Generated new TSK 0
I20260806 04:42:12.285239 14657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:42:12.312665 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 14569
2026-08-06T04:42:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1473 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:42:12 dist-test-slave-vkjp krb5kdc[14671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:42:12 dist-test-slave-vkjp krb5kdc[14671](info): set up 2 sockets
Aug 06 04:42:12 dist-test-slave-vkjp krb5kdc[14671](info): commencing operation
krb5kdc: starting...
W20260806 04:42:13.397670 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:42:13 dist-test-slave-vkjp krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:42:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:42:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:42:13.526500 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:43351
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:43351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:13.643039 14684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:13.643378 14684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:13.643452 14684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:13.647575 14684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:42:13.647679 14684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:13.647719 14684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:13.647774 14684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:42:13.647819 14684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:42:13.653295 14684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:43351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:43351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:13.654413 14684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:13.655470 14684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:13.664630 14693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:13.664862 14684 server_base.cc:1034] running on GCE node
W20260806 04:42:13.664609 14690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:13.664676 14691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:13.665658 14684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:13.666729 14684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:13.667897 14684 hybrid_clock.cc:648] HybridClock initialized: now 1785991333667885 us; error 28 us; skew 500 ppm
Aug 06 04:42:13 dist-test-slave-vkjp krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991333, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:13.675410 14684 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:42:13.676491 14684 webserver.cc:460] Webserver started at http://127.31.166.190:39589/ using document root <none> and password file <none>
I20260806 04:42:13.677098 14684 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:13.677201 14684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:13.677460 14684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:42:13.679718 14684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "1139ba30b3c34d4f83414009cb464323"
format_stamp: "Formatted at 2026-08-06 04:42:13 on dist-test-slave-vkjp"
server_key: "0954a9fa41318455b707c4221c3462a5"
server_key_iv: "1e74aeb1bec0cd2dc5c3b451840dcfcc"
server_key_version: "encryptionkey@0"
I20260806 04:42:13.680351 14684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "1139ba30b3c34d4f83414009cb464323"
format_stamp: "Formatted at 2026-08-06 04:42:13 on dist-test-slave-vkjp"
server_key: "0954a9fa41318455b707c4221c3462a5"
server_key_iv: "1e74aeb1bec0cd2dc5c3b451840dcfcc"
server_key_version: "encryptionkey@0"
I20260806 04:42:13.684398 14684 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:42:13.687042 14701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:13.688405 14684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:42:13.688565 14684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "1139ba30b3c34d4f83414009cb464323"
format_stamp: "Formatted at 2026-08-06 04:42:13 on dist-test-slave-vkjp"
server_key: "0954a9fa41318455b707c4221c3462a5"
server_key_iv: "1e74aeb1bec0cd2dc5c3b451840dcfcc"
server_key_version: "encryptionkey@0"
I20260806 04:42:13.688711 14684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:13.718930 14684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:13.722158 14684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:13.722442 14684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:13.730473 14684 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:43351
I20260806 04:42:13.730779 14755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:43351 every 8 connection(s)
I20260806 04:42:13.731592 14684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:42:13.733587 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 14684
I20260806 04:42:13.733736 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:42:13.734091 32410 external_mini_cluster.cc:1428] Setting key 237e83d06b1bae7f9d2dee08361e488f
I20260806 04:42:13.734560 14756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:13.741533 14756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Bootstrap starting.
Aug 06 04:42:13 dist-test-slave-vkjp krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:13.744231 14756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:13.745244 14756 log.cc:824] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:13.747699 14756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: No bootstrap required, opened a new log
I20260806 04:42:13.749191 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:13.735522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53702 (local address 127.31.166.190:43351)
0806 04:42:13.735933 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:13.735949 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:13.735986 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:42:13.736635 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:13.736651 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:13.736960 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:13.737229 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:13.738187 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:13.739314 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:13.740507 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:13.740966 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:13.741096 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:13.743983 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:13.743989 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:13.744005 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:13.744036 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:13.745971 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:13.746480 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:13.746486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:13.746491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:13.746625 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:13.746942 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:13.746945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:13.746947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:13.747475 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:13.747659 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:13.747967 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 04:42:13.748271 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":137,"threads_started":1}
I20260806 04:42:13.752112 14756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } }
I20260806 04:42:13.752336 14756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:13.752372 14756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1139ba30b3c34d4f83414009cb464323, State: Initialized, Role: FOLLOWER
I20260806 04:42:13.752713 14756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } }
I20260806 04:42:13.752825 14756 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:42:13.752866 14756 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:42:13.752928 14756 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:13.753814 14756 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } }
I20260806 04:42:13.754102 14756 leader_election.cc:302] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1139ba30b3c34d4f83414009cb464323; no voters:
I20260806 04:42:13.754369 14756 leader_election.cc:288] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:42:13.754613 14761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:42:13.754884 14761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 1 LEADER]: Becoming Leader. State: Replica: 1139ba30b3c34d4f83414009cb464323, State: Running, Role: LEADER
I20260806 04:42:13.755266 14761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } }
I20260806 04:42:13.755663 14756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:42:13.757090 14763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1139ba30b3c34d4f83414009cb464323. Latest consensus state: current_term: 1 leader_uuid: "1139ba30b3c34d4f83414009cb464323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } } }
I20260806 04:42:13.757144 14762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1139ba30b3c34d4f83414009cb464323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } } }
I20260806 04:42:13.757215 14763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:13.757239 14762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:13.757558 14771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:42:13.760460 14771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:42:13.766171 14771 catalog_manager.cc:1347] Generated new cluster ID: 488467045d884684a3fbcb93f3c981dd
I20260806 04:42:13.766247 14771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:42:13.792620 14771 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:42:13.793656 14771 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:42:13.797468 14771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Generated new TSK 0
I20260806 04:42:13.798130 14771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:42:13.830075 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43351
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:13.939879 14783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:13.940142 14783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:13.940186 14783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:13.944201 14783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:13.944267 14783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:13.944332 14783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:42:13.949582 14783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:13.950701 14783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:13.951786 14783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:13.961647 14788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:13.961711 14783 server_base.cc:1034] running on GCE node
W20260806 04:42:13.961668 14789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:13.961668 14791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:13.962379 14783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:13.963029 14783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:13.964239 14783 hybrid_clock.cc:648] HybridClock initialized: now 1785991333964221 us; error 29 us; skew 500 ppm
Aug 06 04:42:13 dist-test-slave-vkjp krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991333, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:13.967025 14783 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:42:13.967907 14783 webserver.cc:460] Webserver started at http://127.31.166.129:40625/ using document root <none> and password file <none>
I20260806 04:42:13.968470 14783 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:13.968560 14783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:13.968792 14783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:42:13.970825 14783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "82000fb2de8d4fec86e6c92e635f0983"
format_stamp: "Formatted at 2026-08-06 04:42:13 on dist-test-slave-vkjp"
server_key: "6acd39bfd40d2d11f82863573b7f47d9"
server_key_iv: "00e7d19556f53b608b4442d626b69569"
server_key_version: "encryptionkey@0"
I20260806 04:42:13.971410 14783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "82000fb2de8d4fec86e6c92e635f0983"
format_stamp: "Formatted at 2026-08-06 04:42:13 on dist-test-slave-vkjp"
server_key: "6acd39bfd40d2d11f82863573b7f47d9"
server_key_iv: "00e7d19556f53b608b4442d626b69569"
server_key_version: "encryptionkey@0"
I20260806 04:42:13.975191 14783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20260806 04:42:13.977576 14798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:13.978806 14783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:42:13.978951 14783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "82000fb2de8d4fec86e6c92e635f0983"
format_stamp: "Formatted at 2026-08-06 04:42:13 on dist-test-slave-vkjp"
server_key: "6acd39bfd40d2d11f82863573b7f47d9"
server_key_iv: "00e7d19556f53b608b4442d626b69569"
server_key_version: "encryptionkey@0"
I20260806 04:42:13.979063 14783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:13.992604 14783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:13.995992 14783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:13.996206 14783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:13.996824 14783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:13.997681 14783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:13.997754 14783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:13.997835 14783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:13.997880 14783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:14.008647 14783 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:34241
I20260806 04:42:14.008671 14920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:34241 every 8 connection(s)
I20260806 04:42:14.009662 14783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:42:14.016785 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 14783
I20260806 04:42:14.016906 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:42:14.017241 32410 external_mini_cluster.cc:1428] Setting key 40e71395fe27073bd202497d11556df3
Aug 06 04:42:14 dist-test-slave-vkjp krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991333, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:14.025193 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.011636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:33163 (local address 127.31.166.190:43351)
0806 04:42:14.011806 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:14.011810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:14.012717 (+ 907us) server_negotiation.cc:407] Connection header received
0806 04:42:14.014141 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:14.014144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:14.014197 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:14.014293 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:14.015612 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.016172 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.017533 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.017836 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.017968 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.021062 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:14.021066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:14.021071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:14.021105 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:14.022769 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:14.023457 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:14.023460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:14.023461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:14.023538 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:14.023935 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:14.023937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:14.023938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:14.024135 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:14.024211 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:14.024939 (+ 728us) server_negotiation.cc:299] Negotiation successful
0806 04:42:14.025055 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 04:42:14.026467 14923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.012080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:33163)
0806 04:42:14.012548 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:14.012592 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:14.013916 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:14.014472 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:14.014479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:14.014854 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:14.015440 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.015455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.016345 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.016347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.017412 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.017421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.018014 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.018019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.018224 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.019018 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:14.019035 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:14.020866 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:14.022892 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:14.022901 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:14.022921 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:14.023352 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:14.023636 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:14.023639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:14.023641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:14.023831 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:14.024251 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:14.024261 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:14.024633 (+ 372us) client_negotiation.cc:769] Sending connection context
0806 04:42:14.024870 (+ 237us) client_negotiation.cc:240] Negotiation successful
0806 04:42:14.025290 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260806 04:42:14.027884 14921 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43351
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
I20260806 04:42:14.028133 14921 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:14.028663 14921 heartbeater.cc:511] Master 127.31.166.190:43351 requested a full tablet report, sending...
I20260806 04:42:14.030308 14718 ts_manager.cc:194] Registered new tserver with Master: 82000fb2de8d4fec86e6c92e635f0983 (127.31.166.129:34241)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
I20260806 04:42:14.032267 14718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:33163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:42:14.048327 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43351
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:14.160313 14926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:14.160650 14926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:14.160727 14926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:14.164901 14926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:14.164963 14926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:14.165053 14926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:42:14.170378 14926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:14.171458 14926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:14.172528 14926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:42:14.182407 14926 server_base.cc:1034] running on GCE node
W20260806 04:42:14.182360 14932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:14.182375 14931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:14.182360 14934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:14.183146 14926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:14.183866 14926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:14.185060 14926 hybrid_clock.cc:648] HybridClock initialized: now 1785991334185023 us; error 49 us; skew 500 ppm
Aug 06 04:42:14 dist-test-slave-vkjp krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991334, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:14.187848 14926 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:42:14.188645 14926 webserver.cc:460] Webserver started at http://127.31.166.130:40999/ using document root <none> and password file <none>
I20260806 04:42:14.189152 14926 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:14.189210 14926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:14.189381 14926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:42:14.191438 14926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "12a3522a198e441ebd834fa75527171a"
format_stamp: "Formatted at 2026-08-06 04:42:14 on dist-test-slave-vkjp"
server_key: "bd64048af30d8b62e61dcc5e5db7bae2"
server_key_iv: "ac8c0e4e4e11922f3babce5ed260722e"
server_key_version: "encryptionkey@0"
I20260806 04:42:14.191931 14926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "12a3522a198e441ebd834fa75527171a"
format_stamp: "Formatted at 2026-08-06 04:42:14 on dist-test-slave-vkjp"
server_key: "bd64048af30d8b62e61dcc5e5db7bae2"
server_key_iv: "ac8c0e4e4e11922f3babce5ed260722e"
server_key_version: "encryptionkey@0"
I20260806 04:42:14.195573 14926 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:42:14.197968 14941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:14.199232 14926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:42:14.199362 14926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "12a3522a198e441ebd834fa75527171a"
format_stamp: "Formatted at 2026-08-06 04:42:14 on dist-test-slave-vkjp"
server_key: "bd64048af30d8b62e61dcc5e5db7bae2"
server_key_iv: "ac8c0e4e4e11922f3babce5ed260722e"
server_key_version: "encryptionkey@0"
I20260806 04:42:14.199448 14926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:14.211803 14926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:14.214691 14926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:14.214843 14926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:14.215472 14926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:14.216252 14926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:14.216300 14926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:14.216347 14926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:14.216363 14926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:14.225565 14926 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:45553
I20260806 04:42:14.225593 15062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:45553 every 8 connection(s)
I20260806 04:42:14.226462 14926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:42:14.234994 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 14926
I20260806 04:42:14.235183 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:42:14.235536 32410 external_mini_cluster.cc:1428] Setting key 974e2ea0d927a148cc37e674779d90c8
Aug 06 04:42:14 dist-test-slave-vkjp krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991334, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:14.240967 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.228250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:53717 (local address 127.31.166.190:43351)
0806 04:42:14.228428 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:14.228433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:14.229276 (+ 843us) server_negotiation.cc:407] Connection header received
0806 04:42:14.230598 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:14.230601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:14.230649 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:14.230727 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:14.232176 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.232701 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.233903 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.234142 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.234227 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.237232 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:14.237235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:14.237237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:14.237261 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:14.238859 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:14.239454 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:14.239458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:14.239461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:14.239543 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:14.239903 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:14.239906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:14.239908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:14.240088 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:14.240194 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:14.240676 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 04:42:14.240798 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 04:42:14.241829 15065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.228685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:53717)
0806 04:42:14.229120 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:14.229161 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:14.230394 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:14.230933 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:14.230941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:14.231483 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:14.231986 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.232017 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.232838 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.232841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.233780 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.233792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.234314 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.234317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.234453 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.235250 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:14.235281 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:14.237022 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:14.238980 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:14.238986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:14.239000 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:14.239347 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:14.239638 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:14.239643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:14.239645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:14.239794 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:14.240192 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:14.240199 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:14.240439 (+ 240us) client_negotiation.cc:769] Sending connection context
0806 04:42:14.240620 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 04:42:14.240944 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260806 04:42:14.242991 15063 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43351
I20260806 04:42:14.243279 15063 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:14.243888 15063 heartbeater.cc:511] Master 127.31.166.190:43351 requested a full tablet report, sending...
I20260806 04:42:14.245056 14718 ts_manager.cc:194] Registered new tserver with Master: 12a3522a198e441ebd834fa75527171a (127.31.166.130:45553)
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:42:14.245715 14718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:53717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:42:14.266134 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43351
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:14.391040 15068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:14.391449 15068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:14.391499 15068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:14.396905 15068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:14.397001 15068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:14.397074 15068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:42:14.403426 15068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:14.404582 15068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:14.405675 15068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:14.415889 15076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:14.415899 15073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:14.416040 15074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:14.416419 15068 server_base.cc:1034] running on GCE node
I20260806 04:42:14.416846 15068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:14.417562 15068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:14.418771 15068 hybrid_clock.cc:648] HybridClock initialized: now 1785991334418755 us; error 36 us; skew 500 ppm
Aug 06 04:42:14 dist-test-slave-vkjp krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991334, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:14.421813 15068 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:42:14.422959 15068 webserver.cc:460] Webserver started at http://127.31.166.131:44833/ using document root <none> and password file <none>
I20260806 04:42:14.423678 15068 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:14.423789 15068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:14.424043 15068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:42:14.426247 15068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "c6a463276161489eaac26e2a77da982e"
format_stamp: "Formatted at 2026-08-06 04:42:14 on dist-test-slave-vkjp"
server_key: "2dfd3da988a89673f1543b4f2ab55809"
server_key_iv: "acf6f2e8893dc47d779f327a00a3b57e"
server_key_version: "encryptionkey@0"
I20260806 04:42:14.426856 15068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "c6a463276161489eaac26e2a77da982e"
format_stamp: "Formatted at 2026-08-06 04:42:14 on dist-test-slave-vkjp"
server_key: "2dfd3da988a89673f1543b4f2ab55809"
server_key_iv: "acf6f2e8893dc47d779f327a00a3b57e"
server_key_version: "encryptionkey@0"
I20260806 04:42:14.431033 15068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 04:42:14.433794 15083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:14.435355 15068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:42:14.435524 15068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "c6a463276161489eaac26e2a77da982e"
format_stamp: "Formatted at 2026-08-06 04:42:14 on dist-test-slave-vkjp"
server_key: "2dfd3da988a89673f1543b4f2ab55809"
server_key_iv: "acf6f2e8893dc47d779f327a00a3b57e"
server_key_version: "encryptionkey@0"
I20260806 04:42:14.435662 15068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:14.447713 15068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:14.451233 15068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:14.451483 15068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:14.452112 15068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:14.452996 15068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:14.453070 15068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:14.453145 15068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:14.453197 15068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:14.463330 15068 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:44149
I20260806 04:42:14.463390 15203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:44149 every 8 connection(s)
I20260806 04:42:14.464293 15068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:42:14.474018 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 15068
I20260806 04:42:14.474140 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:42:14.474462 32410 external_mini_cluster.cc:1428] Setting key 07d71783a282bc59db7e1165009f7223
Aug 06 04:42:14 dist-test-slave-vkjp krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991334, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:14.479535 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.466186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:39361 (local address 127.31.166.190:43351)
0806 04:42:14.466380 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:14.466384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:14.467166 (+ 782us) server_negotiation.cc:407] Connection header received
0806 04:42:14.468649 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:14.468652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:14.468743 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:14.468879 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:14.470081 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.470628 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.471868 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.472096 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.472212 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.475397 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:14.475401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:14.475404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:14.475432 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:14.477264 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:14.477795 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:14.477798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:14.477800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:14.477893 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:14.478338 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:14.478341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:14.478342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:14.478545 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:14.478625 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:14.479245 (+ 620us) server_negotiation.cc:299] Negotiation successful
0806 04:42:14.479374 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:42:14.480453 15206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.466499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:39361)
0806 04:42:14.466984 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:14.467028 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:14.468398 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:14.469063 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:14.469070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:14.469439 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:14.469929 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.469942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.470761 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.470764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.471733 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.471743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.472258 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.472262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.472479 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.473156 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:14.473173 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:14.475130 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:14.477401 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:14.477406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:14.477418 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:14.477675 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:14.478005 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:14.478008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:14.478009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:14.478171 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:14.478669 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:14.478675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:14.478926 (+ 251us) client_negotiation.cc:769] Sending connection context
0806 04:42:14.479143 (+ 217us) client_negotiation.cc:240] Negotiation successful
0806 04:42:14.479482 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":126,"threads_started":1}
I20260806 04:42:14.481787 15204 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43351
I20260806 04:42:14.482075 15204 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:14.482796 15204 heartbeater.cc:511] Master 127.31.166.190:43351 requested a full tablet report, sending...
I20260806 04:42:14.484051 14718 ts_manager.cc:194] Registered new tserver with Master: c6a463276161489eaac26e2a77da982e (127.31.166.131:44149)
I20260806 04:42:14.484625 14718 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:39361
I20260806 04:42:14.489579 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:42:14.499890 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.491912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.31.166.190:43351)
0806 04:42:14.492159 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:14.492162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:14.492201 (+ 39us) server_negotiation.cc:407] Connection header received
0806 04:42:14.492237 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:14.492239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:14.492311 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:14.492376 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:14.493240 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.493798 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.494686 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.494938 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.495032 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.496595 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:14.496598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:14.496601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:14.496624 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:14.498138 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:14.498555 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:14.498558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:14.498559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:14.498654 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:14.498953 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:14.498956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:14.498957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:14.499279 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:14.499399 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:14.499642 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 04:42:14.499751 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 04:42:14.503600 14718 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:42:14.506961 14718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:42:14.522590 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.516214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:45553 (local address 127.0.0.1:33482)
0806 04:42:14.516761 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:14.516810 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:14.516970 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:14.517978 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:14.517986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:14.518019 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:14.518172 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.518183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.519493 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.519496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.521067 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.521078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.521988 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.521992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.522099 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.522136 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 04:42:14.522273 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 04:42:14.522406 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":46,"thread_start_us":104,"threads_started":1}
I20260806 04:42:14.522750 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.517462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.31.166.131:44149)
0806 04:42:14.517836 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:14.517858 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:14.517939 (+ 81us) server_negotiation.cc:407] Connection header received
0806 04:42:14.518096 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:14.518101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:14.518288 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:14.518445 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:14.518648 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.519783 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.521270 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.522122 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.522276 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.522410 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:14.522470 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 04:42:14.522569 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":141,"threads_started":1}
I20260806 04:42:14.523211 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.516417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33482 (local address 127.31.166.130:45553)
0806 04:42:14.517173 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:14.517186 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:14.517218 (+ 32us) server_negotiation.cc:407] Connection header received
0806 04:42:14.517302 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:14.517307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:14.517574 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:14.517745 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:14.518316 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.519341 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.521202 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.521849 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.521991 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.522137 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:14.522908 (+ 771us) server_negotiation.cc:299] Negotiation successful
0806 04:42:14.523030 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":124,"threads_started":1}
I20260806 04:42:14.523319 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.517223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49556 (local address 127.31.166.129:34241)
0806 04:42:14.517931 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:14.517971 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:14.518240 (+ 269us) server_negotiation.cc:407] Connection header received
0806 04:42:14.518345 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:14.518353 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:14.518533 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:14.518674 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:14.519137 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.520140 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.521406 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.522193 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.522299 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.522410 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:14.522997 (+ 587us) server_negotiation.cc:299] Negotiation successful
0806 04:42:14.523174 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":132,"threads_started":1}
I20260806 04:42:14.522589 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.516410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:44149 (local address 127.0.0.1:40224)
0806 04:42:14.517851 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:14.517868 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:14.517971 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:14.518429 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:14.518432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:14.518450 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:14.518548 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.518554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.519989 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.519992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.521145 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.521156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.522253 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.522259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.522355 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.522376 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 04:42:14.522433 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 04:42:14.522498 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1377,"thread_start_us":129,"threads_started":1}
I20260806 04:42:14.522897 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.517035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:34241 (local address 127.0.0.1:49556)
0806 04:42:14.518106 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:14.518121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:14.518228 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:14.518846 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:14.518849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:14.518867 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:14.518967 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.518974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.520293 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.520296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.521279 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.521289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.522479 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.522482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.522568 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.522591 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:42:14.522662 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 04:42:14.522742 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"mutex_wait_us":36,"thread_start_us":94,"threads_started":1}
I20260806 04:42:14.526466 14849 tablet_service.cc:1467] Processing CreateTablet for tablet 17fbbbbe6fc441ceb124ee51f92dd148 (DEFAULT_TABLE table=test-table [id=16651636abe94dcd99d29a13e70fc4db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:14.526463 14995 tablet_service.cc:1467] Processing CreateTablet for tablet 17fbbbbe6fc441ceb124ee51f92dd148 (DEFAULT_TABLE table=test-table [id=16651636abe94dcd99d29a13e70fc4db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:14.527637 14849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17fbbbbe6fc441ceb124ee51f92dd148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:14.527647 14995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17fbbbbe6fc441ceb124ee51f92dd148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:14.529407 15135 tablet_service.cc:1467] Processing CreateTablet for tablet 17fbbbbe6fc441ceb124ee51f92dd148 (DEFAULT_TABLE table=test-table [id=16651636abe94dcd99d29a13e70fc4db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:14.530612 15135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17fbbbbe6fc441ceb124ee51f92dd148. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:14.534612 15222 tablet_bootstrap.cc:492] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: Bootstrap starting.
I20260806 04:42:14.536242 15223 tablet_bootstrap.cc:492] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a: Bootstrap starting.
I20260806 04:42:14.536902 15222 tablet_bootstrap.cc:654] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:14.537727 15222 log.cc:824] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:14.539201 15223 tablet_bootstrap.cc:654] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:14.540104 15223 log.cc:824] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:14.540359 15225 tablet_bootstrap.cc:492] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e: Bootstrap starting.
I20260806 04:42:14.540390 15222 tablet_bootstrap.cc:492] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: No bootstrap required, opened a new log
I20260806 04:42:14.540619 15222 ts_tablet_manager.cc:1397] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:42:14.542524 15223 tablet_bootstrap.cc:492] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a: No bootstrap required, opened a new log
I20260806 04:42:14.542788 15223 ts_tablet_manager.cc:1397] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:42:14.543332 15225 tablet_bootstrap.cc:654] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:14.544286 15225 log.cc:824] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:14.544577 15222 raft_consensus.cc:348] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:14.544890 15222 raft_consensus.cc:374] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:14.544993 15222 raft_consensus.cc:729] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82000fb2de8d4fec86e6c92e635f0983, State: Initialized, Role: FOLLOWER
I20260806 04:42:14.545486 15222 consensus_queue.cc:260] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:14.546658 15225 tablet_bootstrap.cc:492] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e: No bootstrap required, opened a new log
I20260806 04:42:14.546921 15222 ts_tablet_manager.cc:1428] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:42:14.546926 15225 ts_tablet_manager.cc:1397] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:42:14.547353 14921 heartbeater.cc:503] Master 127.31.166.190:43351 was elected leader, sending a full tablet report...
I20260806 04:42:14.547111 15223 raft_consensus.cc:348] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:14.547436 15223 raft_consensus.cc:374] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:14.547492 15223 raft_consensus.cc:729] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a3522a198e441ebd834fa75527171a, State: Initialized, Role: FOLLOWER
I20260806 04:42:14.548110 15223 consensus_queue.cc:260] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:14.549386 15223 ts_tablet_manager.cc:1428] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260806 04:42:14.549840 15063 heartbeater.cc:503] Master 127.31.166.190:43351 was elected leader, sending a full tablet report...
I20260806 04:42:14.551374 15225 raft_consensus.cc:348] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:14.551687 15225 raft_consensus.cc:374] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:14.552508 15225 raft_consensus.cc:729] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a463276161489eaac26e2a77da982e, State: Initialized, Role: FOLLOWER
I20260806 04:42:14.553004 15225 consensus_queue.cc:260] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:14.553719 15204 heartbeater.cc:503] Master 127.31.166.190:43351 was elected leader, sending a full tablet report...
I20260806 04:42:14.554088 15225 ts_tablet_manager.cc:1428] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:42:14.566844 15228 raft_consensus.cc:482] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:42:14.567096 15228 raft_consensus.cc:504] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:14.568369 15228 leader_election.cc:288] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12a3522a198e441ebd834fa75527171a (127.31.166.130:45553), c6a463276161489eaac26e2a77da982e (127.31.166.131:44149)
I20260806 04:42:14.572923 15221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.568798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:38019 (local address 127.31.166.131:44149)
0806 04:42:14.568955 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:14.568960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:14.569326 (+ 366us) server_negotiation.cc:407] Connection header received
0806 04:42:14.569507 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:14.569510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:14.569563 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:14.569655 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:14.570072 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.570609 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.571797 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.572221 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.572350 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.572429 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:14.572665 (+ 236us) server_negotiation.cc:299] Negotiation successful
0806 04:42:14.572770 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:42:14.572975 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.568814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:59177 (local address 127.31.166.130:45553)
0806 04:42:14.569106 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:14.569111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:14.569133 (+ 22us) server_negotiation.cc:407] Connection header received
0806 04:42:14.569312 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:14.569316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:14.569381 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:14.569464 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:14.569880 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.570480 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.571646 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.572219 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.572462 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.572517 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:14.572769 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 04:42:14.572863 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260806 04:42:14.572958 15232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.568683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:44149 (local address 127.31.166.129:38019)
0806 04:42:14.569175 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:14.569196 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:14.569384 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:14.569706 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:14.569708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:14.569725 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:14.569927 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.569934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.570740 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.570743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.571666 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.571676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.572363 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.572368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.572484 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.572548 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 04:42:14.572630 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 04:42:14.572695 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":64,"thread_start_us":133,"threads_started":1}
I20260806 04:42:14.572963 15231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.568683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:45553 (local address 127.31.166.129:59177)
0806 04:42:14.569066 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:14.569079 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:14.569176 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:14.569520 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:14.569523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:14.569541 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:14.569692 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.569700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.570596 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.570599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.571503 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:14.571517 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.572461 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.572467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:14.572616 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.572649 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 04:42:14.572732 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 04:42:14.572813 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":1,"thread_start_us":142,"threads_started":1}
I20260806 04:42:14.573853 15156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17fbbbbe6fc441ceb124ee51f92dd148" candidate_uuid: "82000fb2de8d4fec86e6c92e635f0983" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a463276161489eaac26e2a77da982e" is_pre_election: true
I20260806 04:42:14.573887 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17fbbbbe6fc441ceb124ee51f92dd148" candidate_uuid: "82000fb2de8d4fec86e6c92e635f0983" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a3522a198e441ebd834fa75527171a" is_pre_election: true
I20260806 04:42:14.574201 15016 raft_consensus.cc:2454] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82000fb2de8d4fec86e6c92e635f0983 in term 0.
I20260806 04:42:14.574203 15156 raft_consensus.cc:2454] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82000fb2de8d4fec86e6c92e635f0983 in term 0.
I20260806 04:42:14.574783 14799 leader_election.cc:302] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12a3522a198e441ebd834fa75527171a, 82000fb2de8d4fec86e6c92e635f0983, c6a463276161489eaac26e2a77da982e; no voters:
I20260806 04:42:14.575155 15228 raft_consensus.cc:2781] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:42:14.575248 15228 raft_consensus.cc:482] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:42:14.575304 15228 raft_consensus.cc:3037] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:14.576507 15228 raft_consensus.cc:504] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:14.576962 15228 leader_election.cc:288] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 1 election: Requested vote from peers 12a3522a198e441ebd834fa75527171a (127.31.166.130:45553), c6a463276161489eaac26e2a77da982e (127.31.166.131:44149)
I20260806 04:42:14.577462 15016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17fbbbbe6fc441ceb124ee51f92dd148" candidate_uuid: "82000fb2de8d4fec86e6c92e635f0983" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a3522a198e441ebd834fa75527171a"
I20260806 04:42:14.577634 15016 raft_consensus.cc:3037] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:14.579281 15016 raft_consensus.cc:2454] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82000fb2de8d4fec86e6c92e635f0983 in term 1.
I20260806 04:42:14.579306 15156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17fbbbbe6fc441ceb124ee51f92dd148" candidate_uuid: "82000fb2de8d4fec86e6c92e635f0983" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6a463276161489eaac26e2a77da982e"
I20260806 04:42:14.579493 15156 raft_consensus.cc:3037] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:14.579721 14800 leader_election.cc:302] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a3522a198e441ebd834fa75527171a, 82000fb2de8d4fec86e6c92e635f0983; no voters:
I20260806 04:42:14.579928 15228 raft_consensus.cc:2781] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:42:14.580221 15228 raft_consensus.cc:686] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 LEADER]: Becoming Leader. State: Replica: 82000fb2de8d4fec86e6c92e635f0983, State: Running, Role: LEADER
I20260806 04:42:14.580638 15228 consensus_queue.cc:237] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:14.580986 15156 raft_consensus.cc:2454] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82000fb2de8d4fec86e6c92e635f0983 in term 1.
I20260806 04:42:14.584247 14717 catalog_manager.cc:5495] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82000fb2de8d4fec86e6c92e635f0983 (127.31.166.129). New cstate: current_term: 1 leader_uuid: "82000fb2de8d4fec86e6c92e635f0983" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } health_report { overall_health: HEALTHY } } }
I20260806 04:42:14.609035 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.603661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.31.166.129:34241)
0806 04:42:14.603813 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:14.603818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:14.603834 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:42:14.604128 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:14.604136 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:14.604215 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:14.604346 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:42:14.604770 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.605397 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.606613 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:14.607031 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:14.607208 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:14.607793 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:42:14.608493 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:42:14.608658 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:14.608749 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 04:42:14.608828 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:42:14.616369 15156 raft_consensus.cc:1261] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e [term 1 FOLLOWER]: Refusing update from remote peer 82000fb2de8d4fec86e6c92e635f0983: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:42:14.616396 15016 raft_consensus.cc:1261] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a [term 1 FOLLOWER]: Refusing update from remote peer 82000fb2de8d4fec86e6c92e635f0983: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:42:14.617198 15228 consensus_queue.cc:1035] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:42:14.617434 15233 consensus_queue.cc:1035] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:42:14.625694 15236 mvcc.cc:205] Tried to move back new op lower bound from 7315420506579656704 to 7315420506445824000. Current Snapshot: MvccSnapshot[applied={T|T < 7315420506579656704}]
I20260806 04:42:14.636138 14718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:42:14.636337 15238 mvcc.cc:205] Tried to move back new op lower bound from 7315420506579656704 to 7315420506445824000. Current Snapshot: MvccSnapshot[applied={T|T < 7315420506579656704}]
I20260806 04:42:14.636385 14718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:42:14.639379 14718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Sending DeleteTablet for 3 replicas of tablet 17fbbbbe6fc441ceb124ee51f92dd148
I20260806 04:42:14.640291 14849 tablet_service.cc:1514] Processing DeleteTablet for tablet 17fbbbbe6fc441ceb124ee51f92dd148 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:14 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:49556
I20260806 04:42:14.640290 15135 tablet_service.cc:1514] Processing DeleteTablet for tablet 17fbbbbe6fc441ceb124ee51f92dd148 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:14 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:40224
I20260806 04:42:14.640281 14995 tablet_service.cc:1514] Processing DeleteTablet for tablet 17fbbbbe6fc441ceb124ee51f92dd148 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:14 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:33482
I20260806 04:42:14.640761 15134 tablet_service.cc:1514] Processing DeleteTablet for tablet 17fbbbbe6fc441ceb124ee51f92dd148 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:14 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:40224
I20260806 04:42:14.640815 15245 tablet_replica.cc:330] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a: stopping tablet replica
I20260806 04:42:14.641084 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 14783
I20260806 04:42:14.641199 15245 raft_consensus.cc:2229] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:42:14.647404 15247 tablet_replica.cc:330] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e: stopping tablet replica
I20260806 04:42:14.647511 15245 raft_consensus.cc:2258] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:42:14.647835 15247 raft_consensus.cc:2229] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:42:14.648198 15247 raft_consensus.cc:2258] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:42:14.649577 15247 ts_tablet_manager.cc:1905] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:42:14.649602 15245 ts_tablet_manager.cc:1905] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:42:14.652051 15247 ts_tablet_manager.cc:1918] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:42:14.652180 15247 log.cc:1196] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/17fbbbbe6fc441ceb124ee51f92dd148
I20260806 04:42:14.652585 15247 ts_tablet_manager.cc:1939] T 17fbbbbe6fc441ceb124ee51f92dd148 P c6a463276161489eaac26e2a77da982e: Deleting consensus metadata
I20260806 04:42:14.653173 15245 ts_tablet_manager.cc:1918] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:42:14.653330 15245 log.cc:1196] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/17fbbbbe6fc441ceb124ee51f92dd148
I20260806 04:42:14.653728 15245 ts_tablet_manager.cc:1939] T 17fbbbbe6fc441ceb124ee51f92dd148 P 12a3522a198e441ebd834fa75527171a: Deleting consensus metadata
W20260806 04:42:14.653932 14702 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:34241 (error 108)
I20260806 04:42:14.654482 14702 catalog_manager.cc:4841] TS c6a463276161489eaac26e2a77da982e (127.31.166.131:44149): tablet 17fbbbbe6fc441ceb124ee51f92dd148 (table test-table [id=16651636abe94dcd99d29a13e70fc4db]) successfully deleted
W20260806 04:42:14.654816 14702 catalog_manager.cc:4808] TS c6a463276161489eaac26e2a77da982e (127.31.166.131:44149): delete failed for tablet 17fbbbbe6fc441ceb124ee51f92dd148 because the tablet was not found. No further retry: Not found: Tablet not found: 17fbbbbe6fc441ceb124ee51f92dd148
W20260806 04:42:14.655481 15219 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:42:14.655187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:34241 (local address 127.0.0.1:49574)
0806 04:42:14.655327 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:14.655399 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:34241: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":44}
I20260806 04:42:14.655702 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 14926
W20260806 04:42:14.655674 14702 catalog_manager.cc:4584] TS 82000fb2de8d4fec86e6c92e635f0983 (127.31.166.129:34241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17fbbbbe6fc441ceb124ee51f92dd148: Network error: Client connection negotiation failed: client connection to 127.31.166.129:34241: connect: Connection reset by peer (error 104)
I20260806 04:42:14.665009 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 15068
I20260806 04:42:14.666852 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:14.666482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:45553 (local address 127.0.0.1:33484)
0806 04:42:14.666645 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:14.666722 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.130:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 04:42:14.672323 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 14684
I20260806 04:42:14.680702 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:43351
--webserver_interface=127.31.166.190
--webserver_port=39589
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:43351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:14.795351 15249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:14.795650 15249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:14.795692 15249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:14.799896 15249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:42:14.799968 15249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:14.799988 15249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:14.800004 15249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:42:14.800020 15249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:42:14.805435 15249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:43351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:43351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=39589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:14.806558 15249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:14.807639 15249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:14.821144 15258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:14.821153 15255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:14.821345 15249 server_base.cc:1034] running on GCE node
W20260806 04:42:14.821110 15256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:14.822062 15249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:14.823341 15249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:14.824532 15249 hybrid_clock.cc:648] HybridClock initialized: now 1785991334824495 us; error 48 us; skew 500 ppm
Aug 06 04:42:14 dist-test-slave-vkjp krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991334, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:14.827601 15249 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:42:14.828562 15249 webserver.cc:460] Webserver started at http://127.31.166.190:39589/ using document root <none> and password file <none>
I20260806 04:42:14.829160 15249 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:14.829308 15249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:14.833238 15249 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260806 04:42:14.835484 15265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:14.836644 15249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:42:14.836790 15249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "1139ba30b3c34d4f83414009cb464323"
format_stamp: "Formatted at 2026-08-06 04:42:13 on dist-test-slave-vkjp"
server_key: "0954a9fa41318455b707c4221c3462a5"
server_key_iv: "1e74aeb1bec0cd2dc5c3b451840dcfcc"
server_key_version: "encryptionkey@0"
I20260806 04:42:14.837493 15249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:14.854940 15249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:14.858522 15249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:14.858765 15249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:14.866215 15249 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:43351
I20260806 04:42:14.866526 15321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:43351 every 8 connection(s)
I20260806 04:42:14.867453 15249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:42:14.868032 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 15249
I20260806 04:42:14.868602 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:34241
--local_ip_for_outbound_sockets=127.31.166.129
--tserver_master_addrs=127.31.166.190:43351
--webserver_port=40625
--webserver_interface=127.31.166.129
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:42:14.872045 15322 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:42:14.873777 15322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Bootstrap starting.
I20260806 04:42:14.891103 15322 log.cc:824] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:14.899899 15322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:42:14.900306 15322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Bootstrap complete.
I20260806 04:42:14.904562 15322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } }
I20260806 04:42:14.904984 15322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1139ba30b3c34d4f83414009cb464323, State: Initialized, Role: FOLLOWER
I20260806 04:42:14.905458 15322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } }
I20260806 04:42:14.905598 15322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:42:14.905692 15322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:42:14.905824 15322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:42:14.907781 15322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } }
I20260806 04:42:14.908120 15322 leader_election.cc:302] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1139ba30b3c34d4f83414009cb464323; no voters:
I20260806 04:42:14.908430 15322 leader_election.cc:288] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 04:42:14.908588 15326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:42:14.908854 15326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 2 LEADER]: Becoming Leader. State: Replica: 1139ba30b3c34d4f83414009cb464323, State: Running, Role: LEADER
I20260806 04:42:14.909253 15326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } }
I20260806 04:42:14.909677 15322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:42:14.910777 15328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1139ba30b3c34d4f83414009cb464323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } } }
I20260806 04:42:14.910894 15328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:14.911296 15329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1139ba30b3c34d4f83414009cb464323. Latest consensus state: current_term: 2 leader_uuid: "1139ba30b3c34d4f83414009cb464323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } } }
I20260806 04:42:14.911490 15329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:14.911796 15337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:42:14.916778 15337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:42:14.917445 15337 catalog_manager.cc:1259] Loaded cluster ID: 488467045d884684a3fbcb93f3c981dd
I20260806 04:42:14.917512 15337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:42:14.918926 15337 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 04:42:14.923892 15337 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 04:42:14.929595 15337 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:42:14.930434 15337 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Loaded TSK: 0
I20260806 04:42:14.931690 15337 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:42:14.989857 15324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:14.990175 15324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:14.990267 15324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:14.994603 15324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:14.994717 15324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:14.994832 15324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:42:15.000959 15324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:34241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=40625
--tserver_master_addrs=127.31.166.190:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:15.002136 15324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:15.003314 15324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:15.014469 15352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:15.014521 15351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:15.014484 15354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:15.014999 15324 server_base.cc:1034] running on GCE node
I20260806 04:42:15.015446 15324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:15.016215 15324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:15.017427 15324 hybrid_clock.cc:648] HybridClock initialized: now 1785991335017403 us; error 50 us; skew 500 ppm
Aug 06 04:42:15 dist-test-slave-vkjp krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991335, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:15.020326 15324 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:42:15.021292 15324 webserver.cc:460] Webserver started at http://127.31.166.129:40625/ using document root <none> and password file <none>
I20260806 04:42:15.021858 15324 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:15.021934 15324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:15.026073 15324 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:42:15.028585 15362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:15.029973 15324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:42:15.030119 15324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "82000fb2de8d4fec86e6c92e635f0983"
format_stamp: "Formatted at 2026-08-06 04:42:13 on dist-test-slave-vkjp"
server_key: "6acd39bfd40d2d11f82863573b7f47d9"
server_key_iv: "00e7d19556f53b608b4442d626b69569"
server_key_version: "encryptionkey@0"
I20260806 04:42:15.030694 15324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:15.062637 15324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:15.066268 15324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:15.066495 15324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:15.067147 15324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:15.068260 15369 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 04:42:15.072435 15324 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 04:42:15.072515 15324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260806 04:42:15.072597 15324 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 04:42:15.074297 15324 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 04:42:15.074371 15324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 04:42:15.074640 15369 tablet_bootstrap.cc:492] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: Bootstrap starting.
I20260806 04:42:15.085217 15324 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:34241
I20260806 04:42:15.085362 15484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:34241 every 8 connection(s)
I20260806 04:42:15.086330 15324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:42:15.094381 15369 log.cc:824] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:15.096530 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 15324
I20260806 04:42:15.097247 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:45553
--local_ip_for_outbound_sockets=127.31.166.130
--tserver_master_addrs=127.31.166.190:43351
--webserver_port=40999
--webserver_interface=127.31.166.130
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:42:15 dist-test-slave-vkjp krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991335, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:15.103718 15369 tablet_bootstrap.cc:492] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:42:15.104256 15369 tablet_bootstrap.cc:492] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: Bootstrap complete.
I20260806 04:42:15.104387 15487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.088435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:48449 (local address 127.31.166.190:43351)
0806 04:42:15.088842 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:15.088863 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:15.089305 (+ 442us) server_negotiation.cc:407] Connection header received
0806 04:42:15.090840 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:15.090853 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:15.091213 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:15.091489 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:15.093082 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.094073 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.095549 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.096016 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.096140 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.099255 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:15.099260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:15.099272 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:15.099315 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:15.101102 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:15.101649 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:15.101674 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:15.101680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:15.101845 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:15.102187 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:15.102191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:15.102193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:15.102542 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:15.102687 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:15.103232 (+ 545us) server_negotiation.cc:299] Negotiation successful
0806 04:42:15.103505 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":121,"threads_started":1}
I20260806 04:42:15.104400 15488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.088414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:48449)
0806 04:42:15.089074 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:15.089152 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:15.090276 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:15.091615 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:15.091624 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:15.092238 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:15.092884 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.092905 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.094234 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.094238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.095404 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.095416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.096164 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.096168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.096358 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.096968 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:15.096989 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:15.099027 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:15.101235 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:15.101240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:15.101253 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:15.101519 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:15.101945 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:15.101949 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:15.101951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:15.102089 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:15.102663 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:15.102669 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:15.102917 (+ 248us) client_negotiation.cc:769] Sending connection context
0806 04:42:15.103125 (+ 208us) client_negotiation.cc:240] Negotiation successful
0806 04:42:15.103481 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":157,"threads_started":1}
I20260806 04:42:15.105755 15369 ts_tablet_manager.cc:1397] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.004s
I20260806 04:42:15.106590 15485 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43351
I20260806 04:42:15.106840 15485 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:15.107409 15485 heartbeater.cc:511] Master 127.31.166.190:43351 requested a full tablet report, sending...
I20260806 04:42:15.109886 15283 ts_manager.cc:194] Registered new tserver with Master: 82000fb2de8d4fec86e6c92e635f0983 (127.31.166.129:34241)
I20260806 04:42:15.109710 15369 raft_consensus.cc:348] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:15.110374 15369 raft_consensus.cc:729] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82000fb2de8d4fec86e6c92e635f0983, State: Initialized, Role: FOLLOWER
I20260806 04:42:15.110924 15369 consensus_queue.cc:260] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:15.111943 15369 ts_tablet_manager.cc:1428] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 04:42:15.113947 15283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:48449
I20260806 04:42:15.116438 15485 heartbeater.cc:503] Master 127.31.166.190:43351 was elected leader, sending a full tablet report...
Aug 06 04:42:15 dist-test-slave-vkjp krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991334, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for kudu/127.31.166.129@KRBTEST.COM
I20260806 04:42:15.124809 15492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.113071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:34241 (local address 127.0.0.1:49582)
0806 04:42:15.113461 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:15.113509 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:15.113634 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:15.115207 (+ 1573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:15.115213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:15.115638 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:15.115777 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.115785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.117058 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.117065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.118089 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.118100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.118662 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.118666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.118783 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.119454 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:15.119470 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:15.121046 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:15.122953 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:15.122962 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:15.122967 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:15.123277 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:15.123584 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:15.123588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:15.123590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:15.123680 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:15.124126 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:15.124132 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:15.124258 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 04:42:15.124403 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 04:42:15.124620 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
I20260806 04:42:15.124811 15493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.113149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49582 (local address 127.31.166.129:34241)
0806 04:42:15.114746 (+ 1597us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:15.114755 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:15.114774 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:42:15.114835 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:15.114838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:15.115040 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:15.115206 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:15.115937 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.116930 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.118208 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.118537 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.118624 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.121205 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:15.121214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:15.121223 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:15.121244 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:15.122827 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:15.123394 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:15.123407 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:15.123408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:15.123485 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:15.123775 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:15.123778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:15.123780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:15.124006 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:15.124147 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:15.124486 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 04:42:15.124658 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":96,"threads_started":1}
I20260806 04:42:15.125743 15414 tablet_service.cc:1514] Processing DeleteTablet for tablet 17fbbbbe6fc441ceb124ee51f92dd148 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:14 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:49582
I20260806 04:42:15.126564 15494 tablet_replica.cc:330] stopping tablet replica
I20260806 04:42:15.126878 15494 raft_consensus.cc:2229] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:42:15.127189 15494 raft_consensus.cc:2258] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:42:15.128142 15494 ts_tablet_manager.cc:1905] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:42:15.131623 15494 ts_tablet_manager.cc:1918] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:42:15.131704 15494 log.cc:1196] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal/wals/17fbbbbe6fc441ceb124ee51f92dd148
I20260806 04:42:15.131999 15494 ts_tablet_manager.cc:1939] T 17fbbbbe6fc441ceb124ee51f92dd148 P 82000fb2de8d4fec86e6c92e635f0983: Deleting consensus metadata
I20260806 04:42:15.133695 15266 catalog_manager.cc:4841] TS 82000fb2de8d4fec86e6c92e635f0983 (127.31.166.129:34241): tablet 17fbbbbe6fc441ceb124ee51f92dd148 (table test-table [id=16651636abe94dcd99d29a13e70fc4db]) successfully deleted
W20260806 04:42:15.216451 15490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:15.216758 15490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:15.216840 15490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:15.220916 15490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:15.221028 15490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:15.221122 15490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:42:15.226450 15490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:45553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=40999
--tserver_master_addrs=127.31.166.190:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:15.227608 15490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:15.228703 15490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:15.239001 15499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:15.239023 15500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:15.239023 15502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:15.239264 15490 server_base.cc:1034] running on GCE node
I20260806 04:42:15.239866 15490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:15.240613 15490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:15.241771 15490 hybrid_clock.cc:648] HybridClock initialized: now 1785991335241754 us; error 29 us; skew 500 ppm
Aug 06 04:42:15 dist-test-slave-vkjp krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991335, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:15.244865 15490 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:42:15.245786 15490 webserver.cc:460] Webserver started at http://127.31.166.130:40999/ using document root <none> and password file <none>
I20260806 04:42:15.246344 15490 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:15.246426 15490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:15.250644 15490 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:42:15.252971 15509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:15.254454 15490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:42:15.254611 15490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "12a3522a198e441ebd834fa75527171a"
format_stamp: "Formatted at 2026-08-06 04:42:14 on dist-test-slave-vkjp"
server_key: "bd64048af30d8b62e61dcc5e5db7bae2"
server_key_iv: "ac8c0e4e4e11922f3babce5ed260722e"
server_key_version: "encryptionkey@0"
I20260806 04:42:15.255295 15490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:15.298647 15490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:15.302268 15490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:15.302501 15490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:15.303198 15490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:15.304070 15490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:15.304143 15490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:15.304214 15490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:15.304266 15490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:15.314077 15490 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:45553
I20260806 04:42:15.314101 15629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:45553 every 8 connection(s)
I20260806 04:42:15.315254 15490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:42:15.315829 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 15490
I20260806 04:42:15.316309 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:44149
--local_ip_for_outbound_sockets=127.31.166.131
--tserver_master_addrs=127.31.166.190:43351
--webserver_port=44833
--webserver_interface=127.31.166.131
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:42:15 dist-test-slave-vkjp krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991335, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:15.330618 15487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.317449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:34165 (local address 127.31.166.190:43351)
0806 04:42:15.317592 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:15.317597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:15.318799 (+ 1202us) server_negotiation.cc:407] Connection header received
0806 04:42:15.320342 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:15.320388 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:15.320461 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:15.320561 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:15.321775 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.322305 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.323515 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.323755 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.323866 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.326660 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:15.326663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:15.326665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:15.326692 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:15.328418 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:15.328963 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:15.328967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:15.328969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:15.329060 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:15.329418 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:15.329422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:15.329424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:15.329623 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:15.329761 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:15.330286 (+ 525us) server_negotiation.cc:299] Negotiation successful
0806 04:42:15.330439 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:42:15.331627 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.318007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:34165)
0806 04:42:15.318592 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:15.318637 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:15.320084 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:15.320745 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:15.320753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:15.321138 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:15.321619 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.321633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.322438 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.322446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.323388 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.323399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.323920 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.323925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.324094 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.324722 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:15.324739 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:15.326476 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:15.328559 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:15.328565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:15.328580 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:15.328853 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:15.329160 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:15.329163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:15.329165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:15.329309 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:15.329760 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:15.329767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:15.330040 (+ 273us) client_negotiation.cc:769] Sending connection context
0806 04:42:15.330245 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 04:42:15.330604 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":147,"threads_started":1}
I20260806 04:42:15.333037 15630 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43351
I20260806 04:42:15.333295 15630 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:15.333865 15630 heartbeater.cc:511] Master 127.31.166.190:43351 requested a full tablet report, sending...
I20260806 04:42:15.335064 15283 ts_manager.cc:194] Registered new tserver with Master: 12a3522a198e441ebd834fa75527171a (127.31.166.130:45553)
I20260806 04:42:15.336002 15283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:34165
W20260806 04:42:15.436803 15632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:15.437094 15632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:15.437146 15632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:15.441486 15632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:15.441553 15632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:15.441614 15632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:42:15.447233 15632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:44149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=44833
--tserver_master_addrs=127.31.166.190:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:15.448302 15632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:15.449303 15632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:42:15.459110 15632 server_base.cc:1034] running on GCE node
W20260806 04:42:15.459045 15639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:15.459045 15638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:15.459308 15641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:15.459759 15632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:15.460644 15632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:15.461853 15632 hybrid_clock.cc:648] HybridClock initialized: now 1785991335461824 us; error 39 us; skew 500 ppm
Aug 06 04:42:15 dist-test-slave-vkjp krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991335, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:15.464946 15632 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:42:15.465937 15632 webserver.cc:460] Webserver started at http://127.31.166.131:44833/ using document root <none> and password file <none>
I20260806 04:42:15.466480 15632 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:15.466552 15632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:15.470578 15632 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:42:15.472730 15649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:15.474049 15632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:42:15.474161 15632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "c6a463276161489eaac26e2a77da982e"
format_stamp: "Formatted at 2026-08-06 04:42:14 on dist-test-slave-vkjp"
server_key: "2dfd3da988a89673f1543b4f2ab55809"
server_key_iv: "acf6f2e8893dc47d779f327a00a3b57e"
server_key_version: "encryptionkey@0"
I20260806 04:42:15.474738 15632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:15.491374 15632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:15.494624 15632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:15.494860 15632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:15.495575 15632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:15.496476 15632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:15.496526 15632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:15.496592 15632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:15.496624 15632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:15.506330 15632 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:44149
I20260806 04:42:15.506354 15767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:44149 every 8 connection(s)
I20260806 04:42:15.507382 15632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:42:15.514235 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 15632
Aug 06 04:42:15 dist-test-slave-vkjp krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991335, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:15.523118 15487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.509266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:58087 (local address 127.31.166.190:43351)
0806 04:42:15.509417 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:15.509421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:15.510448 (+ 1027us) server_negotiation.cc:407] Connection header received
0806 04:42:15.512132 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:15.512135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:15.512202 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:15.512294 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:15.513533 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.514126 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.515774 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.516023 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.516104 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.519165 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:15.519168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:15.519170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:15.519193 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:15.520883 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:15.521547 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:15.521552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:15.521554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:15.521635 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:15.521991 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:15.521996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:15.521997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:15.522170 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:15.522304 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:15.522781 (+ 477us) server_negotiation.cc:299] Negotiation successful
0806 04:42:15.522906 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58,"spinlock_wait_cycles":768}
I20260806 04:42:15.523936 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.509701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:58087)
0806 04:42:15.510299 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:15.510344 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:15.511902 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:15.512484 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:15.512491 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:15.512872 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:15.513364 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.513378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.514271 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.514279 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.515642 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.515652 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.516192 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.516198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.516465 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.517179 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:15.517196 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:15.518955 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:15.521020 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:15.521026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:15.521042 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:15.521427 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:15.521736 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:15.521739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:15.521741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:15.521894 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:15.522280 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:15.522288 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:15.522550 (+ 262us) client_negotiation.cc:769] Sending connection context
0806 04:42:15.522740 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 04:42:15.523067 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":133,"threads_started":1}
I20260806 04:42:15.525179 15768 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43351
I20260806 04:42:15.525323 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.514984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53716 (local address 127.31.166.190:43351)
0806 04:42:15.515290 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:15.515294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:15.515470 (+ 176us) server_negotiation.cc:407] Connection header received
0806 04:42:15.515595 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:15.515597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:15.515638 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:15.515716 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:15.516813 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.517443 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.518558 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.518830 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.518949 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.520787 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:15.520791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:15.520794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:15.520840 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:15.522485 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:15.522971 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:15.522976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:15.522977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:15.523065 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:15.523692 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:15.523697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:15.523699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:15.523881 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:15.524120 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:15.524313 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 04:42:15.524454 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"spinlock_wait_cycles":10624,"thread_start_us":119,"threads_started":1}
I20260806 04:42:15.525440 15768 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:15.526055 15768 heartbeater.cc:511] Master 127.31.166.190:43351 requested a full tablet report, sending...
I20260806 04:42:15.527264 15282 ts_manager.cc:194] Registered new tserver with Master: c6a463276161489eaac26e2a77da982e (127.31.166.131:44149)
I20260806 04:42:15.527870 15282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:58087
I20260806 04:42:15.539240 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:42:15.544103 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.539912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.31.166.190:43351)
0806 04:42:15.540038 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:15.540041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:15.540252 (+ 211us) server_negotiation.cc:407] Connection header received
0806 04:42:15.540431 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:15.540434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:15.540542 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:15.540631 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:42:15.540895 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.541572 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.542598 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.542843 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.542931 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.543289 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:42:15.543715 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:42:15.543824 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:15.543937 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 04:42:15.543994 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:42:15.544781 15282 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:42:15.548033 15282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:42:15.555816 15414 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb5badcebd4420a8ed99270aa56dff4 (DEFAULT_TABLE table=test-table [id=753ceca2405a40b395e40d9216f10217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:15.556429 15414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb5badcebd4420a8ed99270aa56dff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:15.559396 15369 tablet_bootstrap.cc:492] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983: Bootstrap starting.
I20260806 04:42:15.560282 15492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.554654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:44149 (local address 127.0.0.1:40238)
0806 04:42:15.554822 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:15.554838 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:15.554915 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:15.555880 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:15.555883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:15.555901 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:15.556015 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.556024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.557392 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.557395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.558332 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.558342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.559815 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.559819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.559923 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.559974 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 04:42:15.560072 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 04:42:15.560154 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 04:42:15.560348 15369 tablet_bootstrap.cc:654] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:15.560350 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.554860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.31.166.131:44149)
0806 04:42:15.555421 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:15.555433 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:15.555458 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:42:15.555540 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:15.555546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:15.555721 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:15.555903 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:15.556164 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.557222 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.558453 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.559369 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.559485 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.559623 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:15.560108 (+ 485us) server_negotiation.cc:299] Negotiation successful
0806 04:42:15.560201 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":111,"threads_started":1}
I20260806 04:42:15.561431 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.555002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:45553 (local address 127.0.0.1:33496)
0806 04:42:15.555355 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:15.555368 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:15.555455 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:15.556647 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:15.556650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:15.556669 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:15.556760 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.556766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.558615 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.558618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.559831 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.559839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.560955 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.560959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.561061 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.561084 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:42:15.561162 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 04:42:15.561229 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":116,"threads_started":1}
I20260806 04:42:15.562420 15369 tablet_bootstrap.cc:492] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983: No bootstrap required, opened a new log
I20260806 04:42:15.562496 15369 ts_tablet_manager.cc:1397] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:42:15.562794 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.555197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.31.166.130:45553)
0806 04:42:15.556203 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:15.556216 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:15.556237 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:42:15.556309 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:15.556314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:15.556491 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:15.556631 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:15.557050 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.558447 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.559956 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.560828 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.560951 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.561095 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:15.561192 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 04:42:15.562645 (+ 1453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":130,"threads_started":1}
I20260806 04:42:15.563050 15369 raft_consensus.cc:348] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:15.563208 15369 raft_consensus.cc:374] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:15.563256 15369 raft_consensus.cc:729] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82000fb2de8d4fec86e6c92e635f0983, State: Initialized, Role: FOLLOWER
I20260806 04:42:15.563416 15369 consensus_queue.cc:260] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:15.563864 15369 ts_tablet_manager.cc:1428] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:42:15.563640 15699 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb5badcebd4420a8ed99270aa56dff4 (DEFAULT_TABLE table=test-table [id=753ceca2405a40b395e40d9216f10217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:15.564785 15699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb5badcebd4420a8ed99270aa56dff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:15.565675 15560 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb5badcebd4420a8ed99270aa56dff4 (DEFAULT_TABLE table=test-table [id=753ceca2405a40b395e40d9216f10217]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:15.566772 15560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb5badcebd4420a8ed99270aa56dff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:15.573197 15781 tablet_bootstrap.cc:492] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e: Bootstrap starting.
I20260806 04:42:15.574661 15782 tablet_bootstrap.cc:492] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a: Bootstrap starting.
I20260806 04:42:15.575824 15781 tablet_bootstrap.cc:654] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:15.576781 15781 log.cc:824] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:15.577235 15782 tablet_bootstrap.cc:654] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:15.578027 15782 log.cc:824] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:15.578751 15781 tablet_bootstrap.cc:492] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e: No bootstrap required, opened a new log
I20260806 04:42:15.578991 15781 ts_tablet_manager.cc:1397] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:42:15.579886 15782 tablet_bootstrap.cc:492] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a: No bootstrap required, opened a new log
I20260806 04:42:15.580078 15782 ts_tablet_manager.cc:1397] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:42:15.582084 15781 raft_consensus.cc:348] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:15.582368 15781 raft_consensus.cc:374] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:15.582436 15781 raft_consensus.cc:729] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a463276161489eaac26e2a77da982e, State: Initialized, Role: FOLLOWER
I20260806 04:42:15.582896 15781 consensus_queue.cc:260] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:15.583459 15782 raft_consensus.cc:348] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:15.583626 15768 heartbeater.cc:503] Master 127.31.166.190:43351 was elected leader, sending a full tablet report...
I20260806 04:42:15.583712 15782 raft_consensus.cc:374] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:15.583775 15782 raft_consensus.cc:729] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a3522a198e441ebd834fa75527171a, State: Initialized, Role: FOLLOWER
I20260806 04:42:15.583886 15781 ts_tablet_manager.cc:1428] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 04:42:15.584179 15782 consensus_queue.cc:260] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:15.584869 15630 heartbeater.cc:503] Master 127.31.166.190:43351 was elected leader, sending a full tablet report...
I20260806 04:42:15.585132 15782 ts_tablet_manager.cc:1428] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260806 04:42:15.587707 15486 tablet.cc:2369] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:42:15.758406 15769 tablet.cc:2369] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:42:15.816598 15631 tablet.cc:2369] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:42:15.896992 15786 raft_consensus.cc:482] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:42:15.897284 15786 raft_consensus.cc:504] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:15.898514 15786 leader_election.cc:288] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12a3522a198e441ebd834fa75527171a (127.31.166.130:45553), 82000fb2de8d4fec86e6c92e635f0983 (127.31.166.129:34241)
I20260806 04:42:15.903013 15770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.898870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:45553 (local address 127.31.166.131:51331)
0806 04:42:15.898981 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:15.898997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:15.899159 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:15.899605 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:15.899608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:15.899632 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:15.899778 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.899786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.900733 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.900736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.901901 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.901912 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.902685 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.902688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.902770 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.902790 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 04:42:15.902851 (+ 61us) client_negotiation.cc:240] Negotiation successful
0806 04:42:15.902903 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260806 04:42:15.903124 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.899109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:51331 (local address 127.31.166.130:45553)
0806 04:42:15.899261 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:15.899265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:15.899281 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:42:15.899395 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:15.899401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:15.899462 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:15.899556 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:15.899924 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.900611 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.902054 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.902565 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.902689 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.902742 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:15.902889 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 04:42:15.902959 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 04:42:15.903838 15789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.898975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:33395 (local address 127.31.166.129:34241)
0806 04:42:15.899318 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:15.899321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:15.899784 (+ 463us) server_negotiation.cc:407] Connection header received
0806 04:42:15.899935 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:15.899938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:15.900006 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:15.900128 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:15.900509 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.901131 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.902419 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.903121 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.903242 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.903425 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:15.903657 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 04:42:15.903729 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":136,"threads_started":1}
I20260806 04:42:15.903846 15790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.898870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:34241 (local address 127.31.166.131:33395)
0806 04:42:15.899661 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:15.899676 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:15.899801 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:15.900199 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:15.900202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:15.900223 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:15.900356 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.900363 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.901266 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.901270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.902289 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:15.902296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.903382 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.903386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:15.903491 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.903538 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 04:42:15.903619 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 04:42:15.903687 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":363,"thread_start_us":145,"threads_started":1}
I20260806 04:42:15.903837 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb5badcebd4420a8ed99270aa56dff4" candidate_uuid: "c6a463276161489eaac26e2a77da982e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a3522a198e441ebd834fa75527171a" is_pre_election: true
I20260806 04:42:15.904192 15580 raft_consensus.cc:2454] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6a463276161489eaac26e2a77da982e in term 0.
I20260806 04:42:15.904501 15434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb5badcebd4420a8ed99270aa56dff4" candidate_uuid: "c6a463276161489eaac26e2a77da982e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82000fb2de8d4fec86e6c92e635f0983" is_pre_election: true
I20260806 04:42:15.904780 15434 raft_consensus.cc:2454] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6a463276161489eaac26e2a77da982e in term 0.
I20260806 04:42:15.904759 15651 leader_election.cc:302] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a3522a198e441ebd834fa75527171a, c6a463276161489eaac26e2a77da982e; no voters:
I20260806 04:42:15.905082 15786 raft_consensus.cc:2781] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:42:15.905190 15786 raft_consensus.cc:482] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:42:15.905253 15786 raft_consensus.cc:3037] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:15.906296 15786 raft_consensus.cc:504] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:15.906677 15786 leader_election.cc:288] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [CANDIDATE]: Term 1 election: Requested vote from peers 12a3522a198e441ebd834fa75527171a (127.31.166.130:45553), 82000fb2de8d4fec86e6c92e635f0983 (127.31.166.129:34241)
I20260806 04:42:15.907028 15434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb5badcebd4420a8ed99270aa56dff4" candidate_uuid: "c6a463276161489eaac26e2a77da982e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82000fb2de8d4fec86e6c92e635f0983"
I20260806 04:42:15.907162 15580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb5badcebd4420a8ed99270aa56dff4" candidate_uuid: "c6a463276161489eaac26e2a77da982e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a3522a198e441ebd834fa75527171a"
I20260806 04:42:15.907207 15434 raft_consensus.cc:3037] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:15.907279 15580 raft_consensus.cc:3037] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:15.908528 15580 raft_consensus.cc:2454] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6a463276161489eaac26e2a77da982e in term 1.
I20260806 04:42:15.908529 15434 raft_consensus.cc:2454] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6a463276161489eaac26e2a77da982e in term 1.
I20260806 04:42:15.908797 15651 leader_election.cc:302] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a3522a198e441ebd834fa75527171a, c6a463276161489eaac26e2a77da982e; no voters:
I20260806 04:42:15.908977 15786 raft_consensus.cc:2781] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:42:15.909195 15786 raft_consensus.cc:686] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [term 1 LEADER]: Becoming Leader. State: Replica: c6a463276161489eaac26e2a77da982e, State: Running, Role: LEADER
I20260806 04:42:15.909546 15786 consensus_queue.cc:237] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:15.912684 15283 catalog_manager.cc:5495] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e reported cstate change: term changed from 0 to 1, leader changed from <none> to c6a463276161489eaac26e2a77da982e (127.31.166.131). New cstate: current_term: 1 leader_uuid: "c6a463276161489eaac26e2a77da982e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } health_report { overall_health: UNKNOWN } } }
I20260806 04:42:15.931517 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.927784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.31.166.131:44149)
0806 04:42:15.927927 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:15.927930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:15.927944 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:42:15.928098 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:15.928101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:15.928155 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:15.928242 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:42:15.928536 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.929069 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.929960 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:15.930174 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:15.930252 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:15.930524 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:42:15.930966 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:42:15.931051 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:15.931335 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 04:42:15.931402 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:42:15.937824 15580 raft_consensus.cc:1261] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a [term 1 FOLLOWER]: Refusing update from remote peer c6a463276161489eaac26e2a77da982e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:42:15.937820 15434 raft_consensus.cc:1261] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Refusing update from remote peer c6a463276161489eaac26e2a77da982e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:42:15.938588 15791 consensus_queue.cc:1035] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [LEADER]: Connected to new peer: Peer: permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:42:15.938868 15786 consensus_queue.cc:1035] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:42:15.945561 15795 mvcc.cc:205] Tried to move back new op lower bound from 7315420511993749504 to 7315420511888269312. Current Snapshot: MvccSnapshot[applied={T|T < 7315420511993749504}]
I20260806 04:42:15.945902 15796 mvcc.cc:205] Tried to move back new op lower bound from 7315420511993749504 to 7315420511888269312. Current Snapshot: MvccSnapshot[applied={T|T < 7315420511993749504}]
I20260806 04:42:15.954679 15283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:42:15.954926 15283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:42:15.958019 15283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Sending DeleteTablet for 3 replicas of tablet 8bb5badcebd4420a8ed99270aa56dff4
I20260806 04:42:15.958875 15414 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb5badcebd4420a8ed99270aa56dff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:15 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:49582
I20260806 04:42:15.958967 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 15324
I20260806 04:42:15.959187 15699 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb5badcebd4420a8ed99270aa56dff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:15 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:40238
I20260806 04:42:15.959216 15560 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb5badcebd4420a8ed99270aa56dff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:15 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:33496
I20260806 04:42:15.962656 15805 tablet_replica.cc:330] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a: stopping tablet replica
I20260806 04:42:15.963152 15805 raft_consensus.cc:2229] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:42:15.963593 15805 raft_consensus.cc:2258] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:42:15.964068 15806 tablet_replica.cc:330] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e: stopping tablet replica
I20260806 04:42:15.964409 15806 raft_consensus.cc:2229] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:42:15.964915 15806 raft_consensus.cc:2258] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:42:15.965914 15805 ts_tablet_manager.cc:1905] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:42:15.966547 15806 ts_tablet_manager.cc:1905] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 04:42:15.967749 15266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:34241 (error 108)
I20260806 04:42:15.969039 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 15490
I20260806 04:42:15.970006 15806 ts_tablet_manager.cc:1918] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:42:15.970155 15806 log.cc:1196] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/8bb5badcebd4420a8ed99270aa56dff4
I20260806 04:42:15.970638 15806 ts_tablet_manager.cc:1939] T 8bb5badcebd4420a8ed99270aa56dff4 P c6a463276161489eaac26e2a77da982e: Deleting consensus metadata
I20260806 04:42:15.972385 15266 catalog_manager.cc:4841] TS c6a463276161489eaac26e2a77da982e (127.31.166.131:44149): tablet 8bb5badcebd4420a8ed99270aa56dff4 (table test-table [id=753ceca2405a40b395e40d9216f10217]) successfully deleted
I20260806 04:42:15.977262 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.976800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:34241 (local address 127.0.0.1:49598)
0806 04:42:15.977007 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:15.977116 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:34241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260806 04:42:15.977517 15266 catalog_manager.cc:4584] TS 82000fb2de8d4fec86e6c92e635f0983 (127.31.166.129:34241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bb5badcebd4420a8ed99270aa56dff4: Network error: Client connection negotiation failed: client connection to 127.31.166.129:34241: connect: Connection refused (error 111)
I20260806 04:42:15.982126 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 15632
I20260806 04:42:15.982288 15778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:15.981973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:45553 (local address 127.0.0.1:33500)
0806 04:42:15.982115 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:15.982181 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.130:45553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 04:42:15.989567 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 15249
I20260806 04:42:15.996811 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:43351
--webserver_interface=127.31.166.190
--webserver_port=39589
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:43351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:16.120352 15808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:16.120642 15808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:16.120688 15808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:16.125196 15808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:42:16.125257 15808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:16.125276 15808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:16.125299 15808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:42:16.125352 15808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:42:16.130723 15808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:43351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:43351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=39589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:16.131866 15808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:16.132941 15808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:16.141703 15813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:16.141744 15814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:16.141952 15808 server_base.cc:1034] running on GCE node
W20260806 04:42:16.141682 15816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:16.142560 15808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:16.143859 15808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:16.145061 15808 hybrid_clock.cc:648] HybridClock initialized: now 1785991336145033 us; error 43 us; skew 500 ppm
Aug 06 04:42:16 dist-test-slave-vkjp krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991336, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:16.148195 15808 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:42:16.149217 15808 webserver.cc:460] Webserver started at http://127.31.166.190:39589/ using document root <none> and password file <none>
I20260806 04:42:16.149847 15808 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:16.149946 15808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:16.154047 15808 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:42:16.156450 15823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:16.157644 15808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:42:16.157793 15808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "1139ba30b3c34d4f83414009cb464323"
format_stamp: "Formatted at 2026-08-06 04:42:13 on dist-test-slave-vkjp"
server_key: "0954a9fa41318455b707c4221c3462a5"
server_key_iv: "1e74aeb1bec0cd2dc5c3b451840dcfcc"
server_key_version: "encryptionkey@0"
I20260806 04:42:16.158417 15808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:16.181687 15808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:16.185084 15808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:16.185348 15808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:16.192973 15808 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:43351
I20260806 04:42:16.193279 15878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:43351 every 8 connection(s)
I20260806 04:42:16.194146 15808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:42:16.194525 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 15808
I20260806 04:42:16.194998 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:34241
--local_ip_for_outbound_sockets=127.31.166.129
--tserver_master_addrs=127.31.166.190:43351
--webserver_port=40625
--webserver_interface=127.31.166.129
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:42:16.199549 15879 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:42:16.202314 15879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Bootstrap starting.
I20260806 04:42:16.223709 15879 log.cc:824] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:16.235356 15879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:42:16.235792 15879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323: Bootstrap complete.
I20260806 04:42:16.240043 15879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } }
I20260806 04:42:16.240443 15879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1139ba30b3c34d4f83414009cb464323, State: Initialized, Role: FOLLOWER
I20260806 04:42:16.241014 15879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } }
I20260806 04:42:16.241140 15879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:42:16.241231 15879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:42:16.241349 15879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:42:16.243371 15879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } }
I20260806 04:42:16.243713 15879 leader_election.cc:302] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1139ba30b3c34d4f83414009cb464323; no voters:
I20260806 04:42:16.244069 15879 leader_election.cc:288] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 04:42:16.244277 15883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:42:16.244581 15883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [term 3 LEADER]: Becoming Leader. State: Replica: 1139ba30b3c34d4f83414009cb464323, State: Running, Role: LEADER
I20260806 04:42:16.244968 15883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } }
I20260806 04:42:16.245407 15879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:42:16.246518 15885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1139ba30b3c34d4f83414009cb464323. Latest consensus state: current_term: 3 leader_uuid: "1139ba30b3c34d4f83414009cb464323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } } }
I20260806 04:42:16.246647 15885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:16.246928 15884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "1139ba30b3c34d4f83414009cb464323" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1139ba30b3c34d4f83414009cb464323" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43351 } } }
I20260806 04:42:16.247025 15884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1139ba30b3c34d4f83414009cb464323 [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:16.247124 15893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:42:16.252435 15893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:42:16.253330 15893 catalog_manager.cc:1259] Loaded cluster ID: 488467045d884684a3fbcb93f3c981dd
I20260806 04:42:16.253398 15893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 04:42:16.254477 15893 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff21829e9ad google::LogMessage::Fail() at ??:0
@ 0x7ff2182a188a google::LogMessage::SendToLog() at ??:0
@ 0x7ff21829e4de google::LogMessage::Flush() at ??:0
@ 0x7ff21829fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff222bddfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff222bded2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff222bdb579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff222c1f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff21a4f03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff21a4f0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff21a4f23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff21a4e1512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff21bb256db start_thread at ??:0
@ 0x7ff216d9c61f clone at ??:0
W20260806 04:42:16.352195 15881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:16.352453 15881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:16.352499 15881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:16.356776 15881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:16.356841 15881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:16.356904 15881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:42:16.362376 15881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:34241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=40625
--tserver_master_addrs=127.31.166.190:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:16.363554 15881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:16.364624 15881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:16.374732 15926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:16.374735 15925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:16.374887 15929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:16.374794 15881 server_base.cc:1034] running on GCE node
I20260806 04:42:16.375608 15881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:16.376292 15881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:16.377507 15881 hybrid_clock.cc:648] HybridClock initialized: now 1785991336377488 us; error 40 us; skew 500 ppm
Aug 06 04:42:16 dist-test-slave-vkjp krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991336, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:16.380533 15881 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:42:16.381458 15881 webserver.cc:460] Webserver started at http://127.31.166.129:40625/ using document root <none> and password file <none>
I20260806 04:42:16.382043 15881 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:16.382133 15881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:16.386086 15881 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:42:16.388371 15936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:16.389654 15881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:42:16.389781 15881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "82000fb2de8d4fec86e6c92e635f0983"
format_stamp: "Formatted at 2026-08-06 04:42:13 on dist-test-slave-vkjp"
server_key: "6acd39bfd40d2d11f82863573b7f47d9"
server_key_iv: "00e7d19556f53b608b4442d626b69569"
server_key_version: "encryptionkey@0"
I20260806 04:42:16.390372 15881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:16.423281 15881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:16.426743 15881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:16.426968 15881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:16.427637 15881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:16.428793 15943 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 04:42:16.435159 15881 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 04:42:16.435245 15881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.000s
I20260806 04:42:16.435330 15881 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 04:42:16.437016 15881 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 04:42:16.437091 15881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 04:42:16.437451 15943 tablet_bootstrap.cc:492] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983: Bootstrap starting.
I20260806 04:42:16.448160 15881 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:34241
I20260806 04:42:16.448436 16054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:34241 every 8 connection(s)
I20260806 04:42:16.449254 15881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:42:16.453394 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 15881
I20260806 04:42:16.453830 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:45553
--local_ip_for_outbound_sockets=127.31.166.130
--tserver_master_addrs=127.31.166.190:43351
--webserver_port=40999
--webserver_interface=127.31.166.130
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:42:16.461685 15943 log.cc:824] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:16.470757 15943 tablet_bootstrap.cc:492] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:42:16.471307 15943 tablet_bootstrap.cc:492] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983: Bootstrap complete.
I20260806 04:42:16.472752 15943 ts_tablet_manager.cc:1397] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983: Time spent bootstrapping tablet: real 0.036s user 0.027s sys 0.004s
W20260806 04:42:16.473788 16057 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:42:16.451299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:55495)
0806 04:42:16.451793 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:16.451847 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:16.452675 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:16.472829 (+ 20154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":139,"threads_started":1}
W20260806 04:42:16.474414 16055 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:43351 (0 consecutive failures): Network error: Failed to ping master at 127.31.166.190:43351: Client connection negotiation failed: client connection to 127.31.166.190:43351: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 04:42:16.475237 16057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:16.474928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:36655)
0806 04:42:16.475072 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:16.475172 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 04:42:16.475956 16057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:16.475721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:59405)
0806 04:42:16.475836 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:16.475897 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 04:42:16.475656 15943 raft_consensus.cc:348] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
W20260806 04:42:16.476182 16055 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 04:42:16.476213 15943 raft_consensus.cc:729] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82000fb2de8d4fec86e6c92e635f0983, State: Initialized, Role: FOLLOWER
I20260806 04:42:16.476696 15943 consensus_queue.cc:260] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:16.477694 15943 ts_tablet_manager.cc:1428] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 04:42:16.477778 16057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:16.477538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:35441)
0806 04:42:16.477645 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:16.477696 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 04:42:16.478377 16057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:16.478168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:59875)
0806 04:42:16.478275 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:16.478321 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 04:42:16.576261 16058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:16.576568 16058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:16.576656 16058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:16.580772 16058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:16.580858 16058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:16.580946 16058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:42:16.586328 16058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:45553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=40999
--tserver_master_addrs=127.31.166.190:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:16.587523 16058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:16.588562 16058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:16.598781 16068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:16.598990 16058 server_base.cc:1034] running on GCE node
W20260806 04:42:16.598788 16065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:16.598783 16066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:16.599584 16058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:16.600266 16058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:16.601445 16058 hybrid_clock.cc:648] HybridClock initialized: now 1785991336601415 us; error 43 us; skew 500 ppm
Aug 06 04:42:16 dist-test-slave-vkjp krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991336, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:16.604245 16058 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:42:16.605144 16058 webserver.cc:460] Webserver started at http://127.31.166.130:40999/ using document root <none> and password file <none>
I20260806 04:42:16.605660 16058 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:16.605717 16058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:16.609553 16058 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:42:16.611725 16075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:16.612946 16058 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:42:16.613040 16058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "12a3522a198e441ebd834fa75527171a"
format_stamp: "Formatted at 2026-08-06 04:42:14 on dist-test-slave-vkjp"
server_key: "bd64048af30d8b62e61dcc5e5db7bae2"
server_key_iv: "ac8c0e4e4e11922f3babce5ed260722e"
server_key_version: "encryptionkey@0"
I20260806 04:42:16.613545 16058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:16.630872 16058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:16.634177 16058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:16.634454 16058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:16.635213 16058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:16.636571 16083 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260806 04:42:16.641757 16083 ts_tablet_manager.cc:1852] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260806 04:42:16.642184 16083 ts_tablet_manager.cc:1905] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:42:16.646289 16083 ts_tablet_manager.cc:1918] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:42:16.646409 16083 log.cc:1196] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/8bb5badcebd4420a8ed99270aa56dff4
I20260806 04:42:16.646831 16083 ts_tablet_manager.cc:1939] T 8bb5badcebd4420a8ed99270aa56dff4 P 12a3522a198e441ebd834fa75527171a: Deleting consensus metadata
I20260806 04:42:16.647189 16058 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260806 04:42:16.647277 16058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.001s sys 0.000s
I20260806 04:42:16.647356 16058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:16.647390 16058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:16.656881 16058 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:45553
I20260806 04:42:16.656911 16195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:45553 every 8 connection(s)
I20260806 04:42:16.657745 16058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:42:16.661191 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:16.659930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:33697)
0806 04:42:16.660401 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:16.660514 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
W20260806 04:42:16.661628 16196 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:43351 (0 consecutive failures): Network error: Failed to ping master at 127.31.166.190:43351: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
I20260806 04:42:16.662222 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:16.662023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:56321)
0806 04:42:16.662124 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:16.662170 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 04:42:16.662963 16199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:16.662640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:54837)
0806 04:42:16.662760 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:16.662834 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 04:42:16.663187 16196 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 04:42:16.664541 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 16058
I20260806 04:42:16.665007 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:44149
--local_ip_for_outbound_sockets=127.31.166.131
--tserver_master_addrs=127.31.166.190:43351
--webserver_port=44833
--webserver_interface=127.31.166.131
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:16.782296 16200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:16.782617 16200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:16.782663 16200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:16.786772 16200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:16.786840 16200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:16.786901 16200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:42:16.792541 16200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:44149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=44833
--tserver_master_addrs=127.31.166.190:43351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:16.793751 16200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:16.794884 16200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:16.805200 16209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:16.805241 16206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:16.805195 16205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:16.805572 16200 server_base.cc:1034] running on GCE node
I20260806 04:42:16.806046 16200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:16.806753 16200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:16.807932 16200 hybrid_clock.cc:648] HybridClock initialized: now 1785991336807913 us; error 47 us; skew 500 ppm
Aug 06 04:42:16 dist-test-slave-vkjp krb5kdc[14671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991336, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:16.810702 16200 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:42:16.811767 16200 webserver.cc:460] Webserver started at http://127.31.166.131:44833/ using document root <none> and password file <none>
I20260806 04:42:16.812343 16200 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:16.812445 16200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:16.816534 16200 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:42:16.818900 16216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:16.820320 16200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:42:16.820447 16200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "c6a463276161489eaac26e2a77da982e"
format_stamp: "Formatted at 2026-08-06 04:42:14 on dist-test-slave-vkjp"
server_key: "2dfd3da988a89673f1543b4f2ab55809"
server_key_iv: "acf6f2e8893dc47d779f327a00a3b57e"
server_key_version: "encryptionkey@0"
I20260806 04:42:16.821041 16200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:16.832870 16200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:16.836416 16200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:16.836634 16200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:16.837261 16200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:16.838140 16200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:16.838199 16200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:16.838269 16200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:16.838302 16200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:16.848289 16200 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:44149
I20260806 04:42:16.848317 16334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:44149 every 8 connection(s)
I20260806 04:42:16.849355 16200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:42:16.852335 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 16200
I20260806 04:42:16.853056 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:16.851591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:43837)
0806 04:42:16.852068 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:16.852181 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":115,"threads_started":1}
W20260806 04:42:16.853582 16335 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:43351 (0 consecutive failures): Network error: Failed to ping master at 127.31.166.190:43351: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
I20260806 04:42:16.854220 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:16.853965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:32865)
0806 04:42:16.854112 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:16.854166 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 04:42:16.855026 16337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:16.854825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:33853)
0806 04:42:16.854933 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:16.854977 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 04:42:16.855253 16335 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 04:42:17.479657 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:17.479180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:33799)
0806 04:42:17.479474 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:17.479538 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":78,"threads_started":1}
I20260806 04:42:17.664350 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:17.663908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:54105)
0806 04:42:17.664203 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:17.664260 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":93,"threads_started":1}
I20260806 04:42:17.856248 16341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:17.855824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:56827)
0806 04:42:17.856085 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:17.856145 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":73,"threads_started":1}
I20260806 04:42:17.931700 16342 raft_consensus.cc:482] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:42:17.931959 16342 raft_consensus.cc:504] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:17.933275 16342 leader_election.cc:288] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6a463276161489eaac26e2a77da982e (127.31.166.131:44149), 12a3522a198e441ebd834fa75527171a (127.31.166.130:45553)
Aug 06 04:42:17 dist-test-slave-vkjp krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991336, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.130@KRBTEST.COM
Aug 06 04:42:17 dist-test-slave-vkjp krb5kdc[14671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991336, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.131@KRBTEST.COM
I20260806 04:42:17.947929 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:17.933810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:35163 (local address 127.31.166.130:45553)
0806 04:42:17.934154 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:17.934169 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:17.934212 (+ 43us) server_negotiation.cc:407] Connection header received
0806 04:42:17.934915 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:17.934921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:17.935331 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:17.935910 (+ 579us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:17.937133 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:17.938089 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:17.939480 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:17.939926 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:17.940042 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:17.942894 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:17.942901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:17.942918 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:17.942958 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:17.944852 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:17.945496 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:17.945504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:17.945510 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:17.945662 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:17.946087 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:17.946089 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:17.946090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:17.946615 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:17.946759 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:17.947471 (+ 712us) server_negotiation.cc:299] Negotiation successful
0806 04:42:17.947735 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":110,"threads_started":1}
I20260806 04:42:17.947925 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:17.933533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:45553 (local address 127.31.166.129:35163)
0806 04:42:17.934042 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:17.934058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:17.934178 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:17.935765 (+ 1587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:17.935772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:17.936486 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:17.936934 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:17.936948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:17.938260 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:17.938267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:17.939328 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:17.939340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:17.940061 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:17.940068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:17.940324 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:17.940971 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:17.940987 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:17.942616 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:17.944984 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:17.944992 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:17.945009 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:17.945362 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:17.945775 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:17.945777 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:17.945779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:17.945946 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:17.946753 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:17.946871 (+ 118us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:17.947165 (+ 294us) client_negotiation.cc:769] Sending connection context
0806 04:42:17.947359 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 04:42:17.947699 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":136,"spinlock_wait_cycles":4352,"thread_start_us":99,"threads_started":1}
I20260806 04:42:17.947950 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:17.933596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:58719 (local address 127.31.166.131:44149)
0806 04:42:17.934024 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:17.934056 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:17.934117 (+ 61us) server_negotiation.cc:407] Connection header received
0806 04:42:17.934918 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:17.934924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:17.935378 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:17.936093 (+ 715us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:17.937115 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:17.938198 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:17.939420 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:17.939998 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:17.940190 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:17.943245 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:17.943249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:17.943258 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:17.943285 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:17.945252 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:17.945787 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:17.945793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:17.945797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:17.945938 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:17.946260 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:17.946264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:17.946265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:17.946739 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:17.946958 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:17.947472 (+ 514us) server_negotiation.cc:299] Negotiation successful
0806 04:42:17.947760 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":91,"threads_started":1}
I20260806 04:42:17.947948 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:17.933476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:44149 (local address 127.31.166.129:58719)
0806 04:42:17.933687 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:17.933704 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:17.933835 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:17.935765 (+ 1930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:17.935772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:17.936469 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:17.936932 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:17.936946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:17.938356 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:17.938358 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:17.939284 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:17.939294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:17.940142 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:17.940145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:17.940272 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:17.941033 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:17.941045 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:17.943020 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:17.945377 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:17.945380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:17.945381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:17.945662 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:17.946050 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:17.946053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:17.946055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:17.946150 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:17.946859 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:17.946870 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:17.947162 (+ 292us) client_negotiation.cc:769] Sending connection context
0806 04:42:17.947360 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 04:42:17.947786 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":268,"spinlock_wait_cycles":23808}
W20260806 04:42:17.950526 15938 leader_election.cc:341] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12a3522a198e441ebd834fa75527171a (127.31.166.130:45553): Not found: Tablet not found: 8bb5badcebd4420a8ed99270aa56dff4
W20260806 04:42:17.950740 15937 leader_election.cc:341] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6a463276161489eaac26e2a77da982e (127.31.166.131:44149): Not found: Tablet not found: 8bb5badcebd4420a8ed99270aa56dff4
I20260806 04:42:17.950870 15937 leader_election.cc:302] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82000fb2de8d4fec86e6c92e635f0983; no voters: 12a3522a198e441ebd834fa75527171a, c6a463276161489eaac26e2a77da982e
I20260806 04:42:17.951225 16342 raft_consensus.cc:2726] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:42:18.480898 16346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:18.480402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:49901)
0806 04:42:18.480715 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:18.480797 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":85,"threads_started":1}
I20260806 04:42:18.665856 16347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:18.665308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:39895)
0806 04:42:18.665647 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:18.665712 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":79,"threads_started":1}
I20260806 04:42:18.857510 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:18.857041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:44325)
0806 04:42:18.857355 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:18.857416 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":75,"threads_started":1}
I20260806 04:42:19.482523 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:19.481790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:54525)
0806 04:42:19.482140 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:19.482418 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":84,"threads_started":1}
I20260806 04:42:19.558598 16350 raft_consensus.cc:482] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:42:19.558735 16350 raft_consensus.cc:504] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:19.559291 16350 leader_election.cc:288] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6a463276161489eaac26e2a77da982e (127.31.166.131:44149), 12a3522a198e441ebd834fa75527171a (127.31.166.130:45553)
W20260806 04:42:19.560120 15937 leader_election.cc:341] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6a463276161489eaac26e2a77da982e (127.31.166.131:44149): Not found: Tablet not found: 8bb5badcebd4420a8ed99270aa56dff4
W20260806 04:42:19.560223 15938 leader_election.cc:341] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12a3522a198e441ebd834fa75527171a (127.31.166.130:45553): Not found: Tablet not found: 8bb5badcebd4420a8ed99270aa56dff4
I20260806 04:42:19.560343 15938 leader_election.cc:302] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82000fb2de8d4fec86e6c92e635f0983; no voters: 12a3522a198e441ebd834fa75527171a, c6a463276161489eaac26e2a77da982e
I20260806 04:42:19.560575 16350 raft_consensus.cc:2726] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:42:19.667068 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:19.666623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:47977)
0806 04:42:19.666903 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:19.666961 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":92,"threads_started":1}
I20260806 04:42:19.858826 16352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:19.858356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:38341)
0806 04:42:19.858647 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:19.858716 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
I20260806 04:42:20.483852 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:20.483412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:33165)
0806 04:42:20.483682 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:20.483740 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":90,"threads_started":1}
I20260806 04:42:20.668397 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:20.667897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:47343)
0806 04:42:20.668218 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:20.668280 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":117,"threads_started":1}
I20260806 04:42:20.860167 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:20.859728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:58883)
0806 04:42:20.860011 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:20.860073 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":79,"threads_started":1}
I20260806 04:42:21.485329 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:21.484890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:36283)
0806 04:42:21.485170 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:21.485231 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":106,"threads_started":1}
I20260806 04:42:21.669572 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:21.669162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:38359)
0806 04:42:21.669426 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:21.669480 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260806 04:42:21.861532 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:21.861067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:59731)
0806 04:42:21.861365 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:21.861428 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
I20260806 04:42:22.486689 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:22.486219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:55797)
0806 04:42:22.486540 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:22.486595 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":102,"threads_started":1}
I20260806 04:42:22.670854 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:22.670387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:35419)
0806 04:42:22.670676 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:22.670743 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
I20260806 04:42:22.745095 16362 raft_consensus.cc:482] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:42:22.745252 16362 raft_consensus.cc:504] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:22.745743 16362 leader_election.cc:288] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6a463276161489eaac26e2a77da982e (127.31.166.131:44149), 12a3522a198e441ebd834fa75527171a (127.31.166.130:45553)
W20260806 04:42:22.746649 15937 leader_election.cc:341] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6a463276161489eaac26e2a77da982e (127.31.166.131:44149): Not found: Tablet not found: 8bb5badcebd4420a8ed99270aa56dff4
W20260806 04:42:22.746766 15938 leader_election.cc:341] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12a3522a198e441ebd834fa75527171a (127.31.166.130:45553): Not found: Tablet not found: 8bb5badcebd4420a8ed99270aa56dff4
I20260806 04:42:22.746876 15938 leader_election.cc:302] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82000fb2de8d4fec86e6c92e635f0983; no voters: 12a3522a198e441ebd834fa75527171a, c6a463276161489eaac26e2a77da982e
I20260806 04:42:22.747174 16362 raft_consensus.cc:2726] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:42:22.862769 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:22.862330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:36065)
0806 04:42:22.862620 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:22.862677 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
I20260806 04:42:23.488190 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:23.487668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:60731)
0806 04:42:23.488012 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:23.488082 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":83,"threads_started":1}
I20260806 04:42:23.672228 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:23.671695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:34453)
0806 04:42:23.672064 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:23.672128 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":81,"threads_started":1}
I20260806 04:42:23.864120 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:23.863632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:53343)
0806 04:42:23.863965 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:23.864026 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":77,"threads_started":1}
I20260806 04:42:24.489522 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:24.489081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:42303)
0806 04:42:24.489364 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:24.489425 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260806 04:42:24.673451 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:24.673018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:49763)
0806 04:42:24.673287 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:24.673350 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260806 04:42:24.865518 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:24.865014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:38109)
0806 04:42:24.865348 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:24.865416 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":89,"threads_started":1}
I20260806 04:42:25.490849 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:25.490352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:51459)
0806 04:42:25.490662 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:25.490727 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
I20260806 04:42:25.675802 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:25.674805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:42133)
0806 04:42:25.675401 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:25.675539 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":168,"threads_started":1}
I20260806 04:42:25.867604 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:25.866733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:33821)
0806 04:42:25.867331 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:25.867436 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":140,"threads_started":1}
I20260806 04:42:26.492316 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:26.491863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:51051)
0806 04:42:26.492146 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:26.492204 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":83,"threads_started":1}
I20260806 04:42:26.677624 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:26.676980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:39113)
0806 04:42:26.677372 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:26.677470 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":81,"threads_started":1}
I20260806 04:42:26.869067 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:26.868649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:55651)
0806 04:42:26.868913 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:26.868970 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
I20260806 04:42:27.161577 16378 raft_consensus.cc:482] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:42:27.161716 16378 raft_consensus.cc:504] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a463276161489eaac26e2a77da982e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 44149 } } peers { permanent_uuid: "12a3522a198e441ebd834fa75527171a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45553 } } peers { permanent_uuid: "82000fb2de8d4fec86e6c92e635f0983" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34241 } }
I20260806 04:42:27.162252 16378 leader_election.cc:288] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6a463276161489eaac26e2a77da982e (127.31.166.131:44149), 12a3522a198e441ebd834fa75527171a (127.31.166.130:45553)
W20260806 04:42:27.163354 15937 leader_election.cc:341] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c6a463276161489eaac26e2a77da982e (127.31.166.131:44149): Not found: Tablet not found: 8bb5badcebd4420a8ed99270aa56dff4
W20260806 04:42:27.163497 15938 leader_election.cc:341] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12a3522a198e441ebd834fa75527171a (127.31.166.130:45553): Not found: Tablet not found: 8bb5badcebd4420a8ed99270aa56dff4
I20260806 04:42:27.163595 15938 leader_election.cc:302] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82000fb2de8d4fec86e6c92e635f0983; no voters: 12a3522a198e441ebd834fa75527171a, c6a463276161489eaac26e2a77da982e
I20260806 04:42:27.163784 16378 raft_consensus.cc:2726] T 8bb5badcebd4420a8ed99270aa56dff4 P 82000fb2de8d4fec86e6c92e635f0983 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 04:42:27.493566 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:27.493145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:46347)
0806 04:42:27.493417 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:27.493470 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
I20260806 04:42:27.679244 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:27.678696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:44929)
0806 04:42:27.679010 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:27.679129 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260806 04:42:27.870469 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:27.869997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:47157)
0806 04:42:27.870308 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:27.870372 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260806 04:42:28.494807 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:28.494372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:54335)
0806 04:42:28.494648 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:28.494707 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
I20260806 04:42:28.680624 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:28.680098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:49373)
0806 04:42:28.680463 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:28.680523 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":95,"threads_started":1}
I20260806 04:42:28.871829 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:28.871339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:35313)
0806 04:42:28.871657 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:28.871719 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":107,"threads_started":1}
I20260806 04:42:29.496140 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:29.495570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:43019)
0806 04:42:29.495962 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:29.496034 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":94,"threads_started":1}
I20260806 04:42:29.681952 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:29.681477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:45513)
0806 04:42:29.681764 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:29.681833 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":95,"threads_started":1}
I20260806 04:42:29.873178 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:29.872715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:41023)
0806 04:42:29.873001 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:29.873067 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":102,"threads_started":1}
I20260806 04:42:30.497337 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:30.496923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:45033)
0806 04:42:30.497177 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:30.497237 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260806 04:42:30.683379 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:30.682906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:55973)
0806 04:42:30.683208 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:30.683263 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":105,"threads_started":1}
I20260806 04:42:30.874382 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:30.873972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:60989)
0806 04:42:30.874222 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:30.874279 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
I20260806 04:42:31.499019 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:31.498573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.129:41881)
0806 04:42:31.498857 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:31.498923 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":86,"threads_started":1}
I20260806 04:42:31.684923 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:31.684417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.130:50469)
0806 04:42:31.684731 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:31.684820 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":90,"threads_started":1}
I20260806 04:42:31.860546 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 15881
I20260806 04:42:31.868541 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 16058
I20260806 04:42:31.875857 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:31.875399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43351 (local address 127.31.166.131:36907)
0806 04:42:31.875690 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:31.875756 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:43351: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":94,"threads_started":1}
I20260806 04:42:31.876142 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 16200
2026-08-06T04:42:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19565 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:42:31 dist-test-slave-vkjp krb5kdc[16396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:42:31 dist-test-slave-vkjp krb5kdc[16396](info): set up 2 sockets
Aug 06 04:42:31 dist-test-slave-vkjp krb5kdc[16396](info): commencing operation
krb5kdc: starting...
W20260806 04:42:32.999933 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:42:33 dist-test-slave-vkjp krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:42:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:42:33.128518 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:33009
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:33009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:33.244318 16409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:33.244664 16409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:33.244710 16409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:33.248842 16409 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 04:42:33.248907 16409 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 04:42:33.248922 16409 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 04:42:33.248936 16409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:42:33.248955 16409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:33.248996 16409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:33.249032 16409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:42:33.249053 16409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:42:33.254494 16409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:33009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:33009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:33.255754 16409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:33.256812 16409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:33.265938 16417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:33.266207 16415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:33.266052 16414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:33.266032 16409 server_base.cc:1034] running on GCE node
I20260806 04:42:33.266904 16409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:33.268111 16409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:33.269325 16409 hybrid_clock.cc:648] HybridClock initialized: now 1785991353269303 us; error 32 us; skew 500 ppm
Aug 06 04:42:33 dist-test-slave-vkjp krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991353, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:33.272883 16409 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:42:33.274089 16409 webserver.cc:460] Webserver started at http://127.31.166.190:38217/ using document root <none> and password file <none>
I20260806 04:42:33.274742 16409 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:33.274845 16409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:33.275135 16409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:42:33.277347 16409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "e1997214492e435b9ac7975570c3e62c"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "35ac87b18bd3733756eae49578ab1f20"
server_key_iv: "d3b2453c06b4db0308f151905ec4b305"
server_key_version: "encryptionkey@0"
I20260806 04:42:33.277969 16409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "e1997214492e435b9ac7975570c3e62c"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "35ac87b18bd3733756eae49578ab1f20"
server_key_iv: "d3b2453c06b4db0308f151905ec4b305"
server_key_version: "encryptionkey@0"
I20260806 04:42:33.282003 16409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:42:33.284921 16425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:33.286392 16409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:42:33.286592 16409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "e1997214492e435b9ac7975570c3e62c"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "35ac87b18bd3733756eae49578ab1f20"
server_key_iv: "d3b2453c06b4db0308f151905ec4b305"
server_key_version: "encryptionkey@0"
I20260806 04:42:33.286732 16409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:33.323674 16409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:33.327278 16409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:33.327514 16409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:33.334779 16409 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:33009
I20260806 04:42:33.335127 16480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:33009 every 8 connection(s)
I20260806 04:42:33.335915 16409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:42:33.336323 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 16409
I20260806 04:42:33.336462 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:42:33.336756 32410 external_mini_cluster.cc:1428] Setting key 1f86ad9ba1f9591d7cc0cebf5281350a
I20260806 04:42:33.339207 16481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:33.344960 16481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Bootstrap starting.
Aug 06 04:42:33 dist-test-slave-vkjp krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:33.347922 16481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:33.348695 16481 log.cc:824] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:33.351197 16481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: No bootstrap required, opened a new log
I20260806 04:42:33.351861 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:33.338177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.31.166.190:33009)
0806 04:42:33.338690 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:33.338704 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:33.338739 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:42:33.339580 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:33.339598 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:33.339923 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:33.340264 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:33.341126 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:33.342132 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:33.343020 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:33.343572 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:33.343740 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:33.346624 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:33.346629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:33.346643 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:33.346673 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:33.348652 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:33.349185 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:33.349191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:33.349199 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:33.349352 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:33.349643 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:33.349647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:33.349650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:33.350086 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:33.350226 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:33.350653 (+ 427us) server_negotiation.cc:299] Negotiation successful
0806 04:42:33.351006 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":154,"threads_started":1}
I20260806 04:42:33.355312 16481 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } }
I20260806 04:42:33.355540 16481 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:33.355618 16481 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1997214492e435b9ac7975570c3e62c, State: Initialized, Role: FOLLOWER
I20260806 04:42:33.356022 16481 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } }
I20260806 04:42:33.356142 16481 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:42:33.356226 16481 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:42:33.356318 16481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:33.357275 16481 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } }
I20260806 04:42:33.357614 16481 leader_election.cc:302] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1997214492e435b9ac7975570c3e62c; no voters:
I20260806 04:42:33.357963 16481 leader_election.cc:288] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:42:33.358090 16486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:42:33.358390 16486 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 1 LEADER]: Becoming Leader. State: Replica: e1997214492e435b9ac7975570c3e62c, State: Running, Role: LEADER
I20260806 04:42:33.358804 16486 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } }
I20260806 04:42:33.359287 16481 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:42:33.360598 16488 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1997214492e435b9ac7975570c3e62c. Latest consensus state: current_term: 1 leader_uuid: "e1997214492e435b9ac7975570c3e62c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } } }
I20260806 04:42:33.360744 16488 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:33.361057 16487 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1997214492e435b9ac7975570c3e62c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } } }
I20260806 04:42:33.361160 16487 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: This master's current role is: LEADER
W20260806 04:42:33.364156 16502 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:42:33.364265 16502 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:42:33.364348 16495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:42:33.365049 16495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:42:33.370525 16495 catalog_manager.cc:1347] Generated new cluster ID: 4534b88ffbd04685ac7c020d9ffda2b3
I20260806 04:42:33.370612 16495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:42:33.379021 16495 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:42:33.380033 16495 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:42:33.389497 16495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Generated new TSK 0
I20260806 04:42:33.390228 16495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:42:33.423556 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33009
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:33.538755 16508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:33.539078 16508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:33.539198 16508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:33.543323 16508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:33.543401 16508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:33.543462 16508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:42:33.548888 16508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:33.550050 16508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:33.551059 16508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:33.561446 16515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:33.561553 16514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:33.561451 16517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:33.561965 16508 server_base.cc:1034] running on GCE node
I20260806 04:42:33.562417 16508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:33.563136 16508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:33.564356 16508 hybrid_clock.cc:648] HybridClock initialized: now 1785991353564339 us; error 26 us; skew 500 ppm
Aug 06 04:42:33 dist-test-slave-vkjp krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991353, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:33.567440 16508 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:42:33.568398 16508 webserver.cc:460] Webserver started at http://127.31.166.129:36533/ using document root <none> and password file <none>
I20260806 04:42:33.569032 16508 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:33.569130 16508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:33.569387 16508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:42:33.571581 16508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "1183be6c6cc74149b98167e49fb300d8"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "86278051fd3f42b6989908daa9bda96b"
server_key_iv: "ea0835be23a2910fee03c6d9b33f4709"
server_key_version: "encryptionkey@0"
I20260806 04:42:33.572167 16508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "1183be6c6cc74149b98167e49fb300d8"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "86278051fd3f42b6989908daa9bda96b"
server_key_iv: "ea0835be23a2910fee03c6d9b33f4709"
server_key_version: "encryptionkey@0"
I20260806 04:42:33.576041 16508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:42:33.578563 16524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:33.580113 16508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:42:33.580248 16508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "1183be6c6cc74149b98167e49fb300d8"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "86278051fd3f42b6989908daa9bda96b"
server_key_iv: "ea0835be23a2910fee03c6d9b33f4709"
server_key_version: "encryptionkey@0"
I20260806 04:42:33.580369 16508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:33.609757 16508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:33.612941 16508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:33.613107 16508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:33.613649 16508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:33.614445 16508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:33.614492 16508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:33.614534 16508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:33.614549 16508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:33.624116 16508 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:42471
I20260806 04:42:33.624140 16643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:42471 every 8 connection(s)
I20260806 04:42:33.625046 16508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:42:33.630988 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 16508
I20260806 04:42:33.631141 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:42:33.631456 32410 external_mini_cluster.cc:1428] Setting key ac0daa7bd715689cb2b322f083978341
Aug 06 04:42:33 dist-test-slave-vkjp krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991353, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:33.641290 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:33.626881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:51131 (local address 127.31.166.190:33009)
0806 04:42:33.627052 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:33.627056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:33.627860 (+ 804us) server_negotiation.cc:407] Connection header received
0806 04:42:33.629267 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:33.629271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:33.629355 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:33.629434 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:33.630793 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:33.631691 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:33.633390 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:33.633790 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:33.633914 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:33.636738 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:33.636745 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:33.636751 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:33.636815 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:33.638624 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:33.639292 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:33.639298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:33.639302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:33.639415 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:33.639824 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:33.639831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:33.639834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:33.640119 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:33.640240 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:33.640896 (+ 656us) server_negotiation.cc:299] Negotiation successful
0806 04:42:33.641070 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:42:33.642102 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:33.627209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:51131)
0806 04:42:33.627686 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:33.627732 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:33.628959 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:33.629622 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:33.629629 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:33.630076 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:33.630587 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:33.630602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:33.631879 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:33.631884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:33.633253 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:33.633263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:33.633948 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:33.633951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:33.634096 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:33.634909 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:33.634935 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:33.636554 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:33.638775 (+ 2221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:33.638781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:33.638795 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:33.639183 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:33.639547 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:33.639550 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:33.639552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:33.639701 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:33.640265 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:33.640271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:33.640599 (+ 328us) client_negotiation.cc:769] Sending connection context
0806 04:42:33.640779 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 04:42:33.641088 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
I20260806 04:42:33.643772 16644 heartbeater.cc:347] Connected to a master server at 127.31.166.190:33009
I20260806 04:42:33.644017 16644 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:33.644488 16644 heartbeater.cc:511] Master 127.31.166.190:33009 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
I20260806 04:42:33.646306 16443 ts_manager.cc:194] Registered new tserver with Master: 1183be6c6cc74149b98167e49fb300d8 (127.31.166.129:42471)
I20260806 04:42:33.647832 16443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:51131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:42:33.662245 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33009
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:33.773800 16649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:33.774050 16649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:33.774093 16649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:33.778038 16649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:33.778100 16649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:33.778156 16649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:42:33.783741 16649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:33.784879 16649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:33.785859 16649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:33.795970 16658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:33.796161 16649 server_base.cc:1034] running on GCE node
W20260806 04:42:33.795997 16656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:33.796353 16654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:33.796797 16649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:33.797492 16649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:33.798674 16649 hybrid_clock.cc:648] HybridClock initialized: now 1785991353798640 us; error 44 us; skew 500 ppm
Aug 06 04:42:33 dist-test-slave-vkjp krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991353, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:33.801714 16649 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:42:33.802579 16649 webserver.cc:460] Webserver started at http://127.31.166.130:34321/ using document root <none> and password file <none>
I20260806 04:42:33.803231 16649 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:33.803304 16649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:33.803566 16649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:42:33.805686 16649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "195bf04a5e674522a9648b8213e85fb1"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "a649efed10885a1746402dfcd6b12cf4"
server_key_iv: "eacc6f3e8bed8c6298edca9cbd32a5a9"
server_key_version: "encryptionkey@0"
I20260806 04:42:33.806263 16649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "195bf04a5e674522a9648b8213e85fb1"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "a649efed10885a1746402dfcd6b12cf4"
server_key_iv: "eacc6f3e8bed8c6298edca9cbd32a5a9"
server_key_version: "encryptionkey@0"
I20260806 04:42:33.810362 16649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 04:42:33.812934 16666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:33.814199 16649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:42:33.814317 16649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "195bf04a5e674522a9648b8213e85fb1"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "a649efed10885a1746402dfcd6b12cf4"
server_key_iv: "eacc6f3e8bed8c6298edca9cbd32a5a9"
server_key_version: "encryptionkey@0"
I20260806 04:42:33.814405 16649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:33.840835 16649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:33.844383 16649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:33.844548 16649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:33.845098 16649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:33.845893 16649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:33.845940 16649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:33.845981 16649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:33.845997 16649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:33.855264 16649 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:39161
I20260806 04:42:33.855298 16784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:39161 every 8 connection(s)
I20260806 04:42:33.856119 16649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:42:33.858930 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 16649
I20260806 04:42:33.859025 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:42:33.859431 32410 external_mini_cluster.cc:1428] Setting key 8c63c5c73aa2703d6c6a07d6fc9b06de
Aug 06 04:42:33 dist-test-slave-vkjp krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991353, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:42:33.871402 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:33.857947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:51343 (local address 127.31.166.190:33009)
0806 04:42:33.858084 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:33.858087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:33.859525 (+ 1438us) server_negotiation.cc:407] Connection header received
0806 04:42:33.860642 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:33.860645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:33.860702 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:33.860789 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:33.862306 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:33.862958 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:33.864112 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:33.864352 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:33.864465 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:33.867524 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:33.867527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:33.867529 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:33.867552 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:33.868984 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:33.869653 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:33.869656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:33.869657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:33.869761 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:33.870109 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:33.870112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:33.870114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:33.870304 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:33.870416 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:33.871031 (+ 615us) server_negotiation.cc:299] Negotiation successful
0806 04:42:33.871178 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 04:42:33.872263 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:33.858334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:51343)
0806 04:42:33.859346 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:33.859404 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:33.860429 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:33.860986 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:33.861027 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:33.861463 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:33.862107 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:33.862146 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:33.863114 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:33.863118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:33.863981 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:33.863991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:33.864503 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:33.864507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:33.864681 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:33.865349 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:33.865365 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:33.867342 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:33.869115 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:33.869125 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:33.869143 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:33.869538 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:33.869861 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:33.869864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:33.869866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:33.870005 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:33.870410 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:33.870417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:33.870714 (+ 297us) client_negotiation.cc:769] Sending connection context
0806 04:42:33.870971 (+ 257us) client_negotiation.cc:240] Negotiation successful
0806 04:42:33.871320 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":611,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
I20260806 04:42:33.873400 16785 heartbeater.cc:347] Connected to a master server at 127.31.166.190:33009
I20260806 04:42:33.873649 16785 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:33.874130 16785 heartbeater.cc:511] Master 127.31.166.190:33009 requested a full tablet report, sending...
I20260806 04:42:33.875190 16443 ts_manager.cc:194] Registered new tserver with Master: 195bf04a5e674522a9648b8213e85fb1 (127.31.166.130:39161)
I20260806 04:42:33.875795 16443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:51343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:42:33.890475 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33009
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:34.002177 16790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:34.002507 16790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:34.002586 16790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:34.006882 16790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:34.006986 16790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:34.007122 16790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:42:34.012763 16790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:34.013922 16790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:34.014961 16790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:34.024897 16798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:34.024904 16796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:34.024921 16795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:34.025338 16790 server_base.cc:1034] running on GCE node
I20260806 04:42:34.025686 16790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:34.026378 16790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:34.027583 16790 hybrid_clock.cc:648] HybridClock initialized: now 1785991354027559 us; error 40 us; skew 500 ppm
Aug 06 04:42:34 dist-test-slave-vkjp krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991354, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:34.030450 16790 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:42:34.031419 16790 webserver.cc:460] Webserver started at http://127.31.166.131:38093/ using document root <none> and password file <none>
I20260806 04:42:34.031984 16790 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:34.032078 16790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:34.032320 16790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:42:34.034304 16790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "edee860e107746ac9fb1601036495d9c"
format_stamp: "Formatted at 2026-08-06 04:42:34 on dist-test-slave-vkjp"
server_key: "8460bec3e85e87bb6b80256a9216a66a"
server_key_iv: "ea2672b1f80678242b9410f68727eed9"
server_key_version: "encryptionkey@0"
I20260806 04:42:34.034866 16790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "edee860e107746ac9fb1601036495d9c"
format_stamp: "Formatted at 2026-08-06 04:42:34 on dist-test-slave-vkjp"
server_key: "8460bec3e85e87bb6b80256a9216a66a"
server_key_iv: "ea2672b1f80678242b9410f68727eed9"
server_key_version: "encryptionkey@0"
I20260806 04:42:34.038664 16790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:42:34.041109 16806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:34.042380 16790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:42:34.042536 16790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "edee860e107746ac9fb1601036495d9c"
format_stamp: "Formatted at 2026-08-06 04:42:34 on dist-test-slave-vkjp"
server_key: "8460bec3e85e87bb6b80256a9216a66a"
server_key_iv: "ea2672b1f80678242b9410f68727eed9"
server_key_version: "encryptionkey@0"
I20260806 04:42:34.042655 16790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:34.066015 16790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:34.069351 16790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:34.069571 16790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:34.070184 16790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:34.071017 16790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:34.071085 16790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:34.071194 16790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:34.071244 16790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:34.081355 16790 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:42257
I20260806 04:42:34.081364 16926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:42257 every 8 connection(s)
I20260806 04:42:34.082274 16790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:42:34.087523 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 16790
I20260806 04:42:34.087666 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:42:34.087990 32410 external_mini_cluster.cc:1428] Setting key ae4a94e9c274ad9141aa0f40b83c8c40
Aug 06 04:42:34 dist-test-slave-vkjp krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991354, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:34.097344 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.084314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:40533 (local address 127.31.166.190:33009)
0806 04:42:34.084456 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:34.084460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:34.085565 (+ 1105us) server_negotiation.cc:407] Connection header received
0806 04:42:34.086442 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:34.086444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:34.086495 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:34.086573 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:34.088445 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.089112 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.090624 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.090853 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.091010 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:34.093718 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:34.093721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:34.093723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:34.093749 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:34.095207 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:34.095705 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:34.095708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:34.095709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:34.095799 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:34.096147 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:34.096150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:34.096151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:34.096418 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:34.096520 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:34.097047 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 04:42:34.097160 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:42:34.098156 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.084692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:40533)
0806 04:42:34.085418 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:34.085459 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:34.086229 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:34.086918 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:34.086925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:34.087514 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:34.088226 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:34.088249 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.089245 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.089251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:34.090451 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:34.090462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.091069 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.091072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:34.091279 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:34.091935 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:34.091951 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:34.093535 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:34.095315 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:34.095321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:34.095332 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:34.095583 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:34.095888 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:34.095890 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:34.095892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:34.096026 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:34.096556 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:34.096561 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:34.096820 (+ 259us) client_negotiation.cc:769] Sending connection context
0806 04:42:34.096990 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 04:42:34.097257 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":107,"threads_started":1}
I20260806 04:42:34.099669 16928 heartbeater.cc:347] Connected to a master server at 127.31.166.190:33009
I20260806 04:42:34.099948 16928 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:34.100502 16928 heartbeater.cc:511] Master 127.31.166.190:33009 requested a full tablet report, sending...
I20260806 04:42:34.101863 16443 ts_manager.cc:194] Registered new tserver with Master: edee860e107746ac9fb1601036495d9c (127.31.166.131:42257)
I20260806 04:42:34.102478 16443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:40533
I20260806 04:42:34.102627 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:42:34.112339 16484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.104908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.31.166.190:33009)
0806 04:42:34.105028 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:34.105038 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:34.105245 (+ 207us) server_negotiation.cc:407] Connection header received
0806 04:42:34.105339 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:34.105341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:34.105380 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:34.105437 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:34.106101 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.106604 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.107548 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.107805 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.107900 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:34.109278 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:34.109281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:34.109283 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:34.109306 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:34.110767 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:34.111236 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:34.111239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:34.111241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:34.111327 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:34.111572 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:34.111574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:34.111575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:34.111807 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:34.111920 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:34.112121 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 04:42:34.112219 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:42:34.115345 16443 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:42:34.118394 16443 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:42:34.133102 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.127910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.31.166.131:42257)
0806 04:42:34.128828 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:34.128857 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:34.128880 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:42:34.128949 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:34.128954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:34.129144 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:34.129295 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:34.129542 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.130419 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.131883 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.132424 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.132560 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:34.132693 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:34.132862 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 04:42:34.132959 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":98,"threads_started":1}
I20260806 04:42:34.133093 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.127724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:42257 (local address 127.0.0.1:57296)
0806 04:42:34.128415 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:34.128428 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:34.128520 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:34.129287 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:34.129290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:34.129307 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:34.129418 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:34.129428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.130573 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.130576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:34.131757 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:34.131764 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.132547 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.132551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:34.132665 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:34.132706 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 04:42:34.132817 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 04:42:34.132940 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260806 04:42:34.133288 16941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.126923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:42471 (local address 127.0.0.1:44450)
0806 04:42:34.127427 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:34.127474 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:34.127613 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:34.128445 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:34.128452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:34.128477 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:34.128622 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:34.128628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.130107 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.130111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:34.131451 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:34.131463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.132762 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.132766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:34.132894 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:34.132977 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 04:42:34.133051 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 04:42:34.133150 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":47,"thread_start_us":74,"threads_started":1}
I20260806 04:42:34.133380 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.127725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.31.166.130:39161)
0806 04:42:34.128455 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:34.128471 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:34.128492 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:42:34.128563 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:34.128567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:34.128747 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:34.128901 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:34.129140 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.130195 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.131750 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.132531 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.132634 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:34.132731 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:34.133146 (+ 415us) server_negotiation.cc:299] Negotiation successful
0806 04:42:34.133235 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":126,"threads_started":1}
I20260806 04:42:34.133675 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.127004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:39161 (local address 127.0.0.1:49666)
0806 04:42:34.128116 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:34.128130 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:34.128212 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:34.128895 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:34.128899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:34.128918 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:34.129018 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:34.129024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.130321 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.130326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:34.131624 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:34.131635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.132779 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.132782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:34.132977 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:34.132995 (+ 18us) client_negotiation.cc:769] Sending connection context
0806 04:42:34.133057 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 04:42:34.133556 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"mutex_wait_us":533,"thread_start_us":127,"threads_started":1}
I20260806 04:42:34.133700 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.127046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.31.166.129:42471)
0806 04:42:34.127450 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:34.127470 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:34.127574 (+ 104us) server_negotiation.cc:407] Connection header received
0806 04:42:34.127842 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:34.127851 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:34.128073 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:34.128238 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:34.128757 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.129967 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.131560 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.132427 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.132553 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:34.132694 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:34.133472 (+ 778us) server_negotiation.cc:299] Negotiation successful
0806 04:42:34.133566 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":115,"threads_started":1}
I20260806 04:42:34.136327 16855 tablet_service.cc:1467] Processing CreateTablet for tablet 6f71c41e44e74fd9b495d2086c71365e (DEFAULT_TABLE table=test-table [id=10c35abf297a4eaa83c2ff1ca51e3cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:34.136459 16575 tablet_service.cc:1467] Processing CreateTablet for tablet 6f71c41e44e74fd9b495d2086c71365e (DEFAULT_TABLE table=test-table [id=10c35abf297a4eaa83c2ff1ca51e3cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:34.136937 16715 tablet_service.cc:1467] Processing CreateTablet for tablet 6f71c41e44e74fd9b495d2086c71365e (DEFAULT_TABLE table=test-table [id=10c35abf297a4eaa83c2ff1ca51e3cfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:34.137373 16855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f71c41e44e74fd9b495d2086c71365e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:34.137472 16575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f71c41e44e74fd9b495d2086c71365e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:34.137797 16715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f71c41e44e74fd9b495d2086c71365e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:34.144284 16947 tablet_bootstrap.cc:492] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c: Bootstrap starting.
I20260806 04:42:34.144783 16948 tablet_bootstrap.cc:492] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: Bootstrap starting.
I20260806 04:42:34.146682 16948 tablet_bootstrap.cc:654] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:34.147156 16947 tablet_bootstrap.cc:654] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:34.147493 16949 tablet_bootstrap.cc:492] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1: Bootstrap starting.
I20260806 04:42:34.147671 16948 log.cc:824] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:34.147964 16947 log.cc:824] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:34.149843 16949 tablet_bootstrap.cc:654] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:34.150121 16948 tablet_bootstrap.cc:492] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: No bootstrap required, opened a new log
I20260806 04:42:34.150324 16948 ts_tablet_manager.cc:1397] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:42:34.150626 16949 log.cc:824] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:34.150876 16947 tablet_bootstrap.cc:492] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c: No bootstrap required, opened a new log
I20260806 04:42:34.151060 16947 ts_tablet_manager.cc:1397] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260806 04:42:34.152277 16949 tablet_bootstrap.cc:492] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1: No bootstrap required, opened a new log
I20260806 04:42:34.152477 16949 ts_tablet_manager.cc:1397] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:42:34.154917 16947 raft_consensus.cc:348] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } } peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } }
I20260806 04:42:34.154917 16948 raft_consensus.cc:348] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } } peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } }
I20260806 04:42:34.155224 16947 raft_consensus.cc:374] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:34.155224 16948 raft_consensus.cc:374] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:34.155287 16948 raft_consensus.cc:729] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1183be6c6cc74149b98167e49fb300d8, State: Initialized, Role: FOLLOWER
I20260806 04:42:34.155321 16947 raft_consensus.cc:729] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edee860e107746ac9fb1601036495d9c, State: Initialized, Role: FOLLOWER
I20260806 04:42:34.155226 16949 raft_consensus.cc:348] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } } peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } }
I20260806 04:42:34.155478 16949 raft_consensus.cc:374] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:34.155576 16949 raft_consensus.cc:729] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195bf04a5e674522a9648b8213e85fb1, State: Initialized, Role: FOLLOWER
I20260806 04:42:34.155807 16948 consensus_queue.cc:260] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } } peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } }
I20260806 04:42:34.155805 16947 consensus_queue.cc:260] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } } peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } }
I20260806 04:42:34.156060 16949 consensus_queue.cc:260] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } } peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } }
I20260806 04:42:34.156870 16948 ts_tablet_manager.cc:1428] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:42:34.156863 16947 ts_tablet_manager.cc:1428] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260806 04:42:34.157059 16949 ts_tablet_manager.cc:1428] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 04:42:34.157222 16644 heartbeater.cc:503] Master 127.31.166.190:33009 was elected leader, sending a full tablet report...
I20260806 04:42:34.157222 16928 heartbeater.cc:503] Master 127.31.166.190:33009 was elected leader, sending a full tablet report...
I20260806 04:42:34.157377 16785 heartbeater.cc:503] Master 127.31.166.190:33009 was elected leader, sending a full tablet report...
I20260806 04:42:34.172053 16953 raft_consensus.cc:482] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:42:34.172323 16953 raft_consensus.cc:504] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } } peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } }
I20260806 04:42:34.173592 16953 leader_election.cc:288] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 195bf04a5e674522a9648b8213e85fb1 (127.31.166.130:39161), edee860e107746ac9fb1601036495d9c (127.31.166.131:42257)
I20260806 04:42:34.177814 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.173906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:47897 (local address 127.31.166.130:39161)
0806 04:42:34.174049 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:34.174056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:34.174374 (+ 318us) server_negotiation.cc:407] Connection header received
0806 04:42:34.174520 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:34.174523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:34.174581 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:34.174667 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:34.175063 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.175795 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.176861 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.177258 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.177395 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:34.177448 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:34.177632 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 04:42:34.177699 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 04:42:34.177846 16956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.173794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:39161 (local address 127.31.166.129:47897)
0806 04:42:34.174225 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:34.174253 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:34.174386 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:34.174697 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:34.174700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:34.174742 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:34.174936 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:34.174943 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.175905 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.175910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:34.176729 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:34.176739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.177395 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.177399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:34.177508 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:34.177527 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:42:34.177615 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 04:42:34.177681 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":56,"spinlock_wait_cycles":28800,"thread_start_us":127,"threads_started":1}
I20260806 04:42:34.178263 16957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.173892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:42257 (local address 127.31.166.129:44217)
0806 04:42:34.174339 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:34.174353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:34.174448 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:34.174697 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:34.174700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:34.174767 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:34.174927 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:34.174936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.176231 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.176236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:34.177010 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:34.177022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.177693 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.177697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:34.177807 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:34.177837 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 04:42:34.178017 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 04:42:34.178094 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":5,"spinlock_wait_cycles":6656,"thread_start_us":105,"threads_started":1}
I20260806 04:42:34.178498 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.174003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:44217 (local address 127.31.166.131:42257)
0806 04:42:34.174181 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:34.174184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:34.174453 (+ 269us) server_negotiation.cc:407] Connection header received
0806 04:42:34.174542 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:34.174545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:34.174588 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:34.174666 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:34.175442 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.176107 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.177149 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.177565 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.177702 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:34.177757 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:34.178257 (+ 500us) server_negotiation.cc:299] Negotiation successful
0806 04:42:34.178350 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 04:42:34.178505 16737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f71c41e44e74fd9b495d2086c71365e" candidate_uuid: "1183be6c6cc74149b98167e49fb300d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195bf04a5e674522a9648b8213e85fb1" is_pre_election: true
I20260806 04:42:34.178849 16737 raft_consensus.cc:2454] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1183be6c6cc74149b98167e49fb300d8 in term 0.
I20260806 04:42:34.179106 16878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f71c41e44e74fd9b495d2086c71365e" candidate_uuid: "1183be6c6cc74149b98167e49fb300d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edee860e107746ac9fb1601036495d9c" is_pre_election: true
I20260806 04:42:34.179365 16526 leader_election.cc:302] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1183be6c6cc74149b98167e49fb300d8, 195bf04a5e674522a9648b8213e85fb1; no voters:
I20260806 04:42:34.179404 16878 raft_consensus.cc:2454] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1183be6c6cc74149b98167e49fb300d8 in term 0.
I20260806 04:42:34.179654 16953 raft_consensus.cc:2781] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:42:34.179795 16953 raft_consensus.cc:482] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:42:34.179867 16953 raft_consensus.cc:3037] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:34.180970 16953 raft_consensus.cc:504] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } } peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } }
I20260806 04:42:34.181356 16953 leader_election.cc:288] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [CANDIDATE]: Term 1 election: Requested vote from peers 195bf04a5e674522a9648b8213e85fb1 (127.31.166.130:39161), edee860e107746ac9fb1601036495d9c (127.31.166.131:42257)
I20260806 04:42:34.181730 16878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f71c41e44e74fd9b495d2086c71365e" candidate_uuid: "1183be6c6cc74149b98167e49fb300d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edee860e107746ac9fb1601036495d9c"
I20260806 04:42:34.181730 16737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f71c41e44e74fd9b495d2086c71365e" candidate_uuid: "1183be6c6cc74149b98167e49fb300d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "195bf04a5e674522a9648b8213e85fb1"
I20260806 04:42:34.181866 16737 raft_consensus.cc:3037] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:34.181866 16878 raft_consensus.cc:3037] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:34.183247 16737 raft_consensus.cc:2454] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1183be6c6cc74149b98167e49fb300d8 in term 1.
I20260806 04:42:34.183256 16878 raft_consensus.cc:2454] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1183be6c6cc74149b98167e49fb300d8 in term 1.
I20260806 04:42:34.183573 16526 leader_election.cc:302] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1183be6c6cc74149b98167e49fb300d8, 195bf04a5e674522a9648b8213e85fb1, edee860e107746ac9fb1601036495d9c; no voters:
I20260806 04:42:34.183728 16953 raft_consensus.cc:2781] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:42:34.183950 16953 raft_consensus.cc:686] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 1 LEADER]: Becoming Leader. State: Replica: 1183be6c6cc74149b98167e49fb300d8, State: Running, Role: LEADER
I20260806 04:42:34.184355 16953 consensus_queue.cc:237] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } } peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } }
I20260806 04:42:34.187791 16441 catalog_manager.cc:5495] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1183be6c6cc74149b98167e49fb300d8 (127.31.166.129). New cstate: current_term: 1 leader_uuid: "1183be6c6cc74149b98167e49fb300d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } health_report { overall_health: UNKNOWN } } }
I20260806 04:42:34.212399 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.208767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44464 (local address 127.31.166.129:42471)
0806 04:42:34.208902 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:34.208906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:34.208922 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:42:34.209044 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:34.209047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:34.209116 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:34.209226 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:42:34.209486 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.210001 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.210891 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:34.211195 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:34.211297 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:34.211634 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:42:34.212028 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:42:34.212164 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:34.212206 (+ 42us) server_negotiation.cc:299] Negotiation successful
0806 04:42:34.212261 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260806 04:42:34.218510 16737 raft_consensus.cc:1261] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Refusing update from remote peer 1183be6c6cc74149b98167e49fb300d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:42:34.218510 16878 raft_consensus.cc:1261] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c [term 1 FOLLOWER]: Refusing update from remote peer 1183be6c6cc74149b98167e49fb300d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:42:34.219452 16953 consensus_queue.cc:1035] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:42:34.219681 16958 consensus_queue.cc:1035] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:42:34.229207 16964 mvcc.cc:205] Tried to move back new op lower bound from 7315420586870681600 to 7315420586742550528. Current Snapshot: MvccSnapshot[applied={T|T < 7315420586870681600}]
I20260806 04:42:34.230446 16963 mvcc.cc:205] Tried to move back new op lower bound from 7315420586870681600 to 7315420586742550528. Current Snapshot: MvccSnapshot[applied={T|T < 7315420586870681600}]
I20260806 04:42:34.236037 16443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:42:34.236191 16443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:42:34.238759 16443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Sending DeleteTablet for 3 replicas of tablet 6f71c41e44e74fd9b495d2086c71365e
I20260806 04:42:34.239630 16575 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f71c41e44e74fd9b495d2086c71365e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:34 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:44450
I20260806 04:42:34.239689 16715 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f71c41e44e74fd9b495d2086c71365e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:34 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:49666
I20260806 04:42:34.240108 16972 tablet_replica.cc:330] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: stopping tablet replica
I20260806 04:42:34.240312 16855 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f71c41e44e74fd9b495d2086c71365e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:34 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:57296
I20260806 04:42:34.240465 16972 raft_consensus.cc:2229] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:42:34.240824 16974 tablet_replica.cc:330] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c: stopping tablet replica
I20260806 04:42:34.240841 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 16508
I20260806 04:42:34.240849 16972 raft_consensus.cc:2258] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:42:34.242105 16974 raft_consensus.cc:2229] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:42:34.242484 16974 raft_consensus.cc:2258] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:42:34.243809 16973 tablet_replica.cc:330] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1: stopping tablet replica
I20260806 04:42:34.244084 16973 raft_consensus.cc:2229] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:42:34.244362 16973 raft_consensus.cc:2258] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:42:34.245927 16974 ts_tablet_manager.cc:1905] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:42:34.245927 16973 ts_tablet_manager.cc:1905] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:42:34.248919 16974 ts_tablet_manager.cc:1918] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:42:34.248919 16973 ts_tablet_manager.cc:1918] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:42:34.249030 16973 log.cc:1196] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/6f71c41e44e74fd9b495d2086c71365e
I20260806 04:42:34.249073 16974 log.cc:1196] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/6f71c41e44e74fd9b495d2086c71365e
I20260806 04:42:34.249478 16973 ts_tablet_manager.cc:1939] T 6f71c41e44e74fd9b495d2086c71365e P 195bf04a5e674522a9648b8213e85fb1: Deleting consensus metadata
I20260806 04:42:34.249500 16974 ts_tablet_manager.cc:1939] T 6f71c41e44e74fd9b495d2086c71365e P edee860e107746ac9fb1601036495d9c: Deleting consensus metadata
W20260806 04:42:34.249521 16426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:42471 (error 108)
I20260806 04:42:34.249805 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 16649
I20260806 04:42:34.250854 16426 catalog_manager.cc:4841] TS edee860e107746ac9fb1601036495d9c (127.31.166.131:42257): tablet 6f71c41e44e74fd9b495d2086c71365e (table test-table [id=10c35abf297a4eaa83c2ff1ca51e3cfb]) successfully deleted
I20260806 04:42:34.251137 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.250672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:42471 (local address 127.0.0.1:44466)
0806 04:42:34.250926 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:34.251011 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260806 04:42:34.251312 16426 catalog_manager.cc:4584] TS 1183be6c6cc74149b98167e49fb300d8 (127.31.166.129:42471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f71c41e44e74fd9b495d2086c71365e: Network error: Client connection negotiation failed: client connection to 127.31.166.129:42471: connect: Connection refused (error 111)
I20260806 04:42:34.256884 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 16790
I20260806 04:42:34.258419 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:34.257060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:39161 (local address 127.0.0.1:49676)
0806 04:42:34.258196 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:34.258261 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.130:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1032}
I20260806 04:42:34.264211 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 16409
I20260806 04:42:54.271951 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:33009
--webserver_interface=127.31.166.190
--webserver_port=38217
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:33009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:42:54.382160 16976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:54.382457 16976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:54.382499 16976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:54.386726 16976 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 04:42:54.386799 16976 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 04:42:54.386826 16976 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 04:42:54.386873 16976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:42:54.386899 16976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:54.386933 16976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:54.386961 16976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:42:54.386996 16976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:42:54.392609 16976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:33009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:33009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=38217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:54.393698 16976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:54.394767 16976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:54.408110 16984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:54.408154 16986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:54.408066 16983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:54.408236 16976 server_base.cc:1034] running on GCE node
I20260806 04:42:54.409047 16976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:54.410193 16976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:54.411399 16976 hybrid_clock.cc:648] HybridClock initialized: now 1785991374411375 us; error 42 us; skew 500 ppm
Aug 06 04:42:54 dist-test-slave-vkjp krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991374, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:54.419427 16976 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:42:54.420506 16976 webserver.cc:460] Webserver started at http://127.31.166.190:38217/ using document root <none> and password file <none>
I20260806 04:42:54.421240 16976 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:54.421372 16976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:54.425765 16976 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:42:54.428122 16993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:54.429543 16976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:42:54.429785 16976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "e1997214492e435b9ac7975570c3e62c"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "35ac87b18bd3733756eae49578ab1f20"
server_key_iv: "d3b2453c06b4db0308f151905ec4b305"
server_key_version: "encryptionkey@0"
I20260806 04:42:54.430404 16976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:54.440192 16976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:54.443519 16976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:54.443727 16976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:54.451067 16976 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:33009
I20260806 04:42:54.451375 17049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:33009 every 8 connection(s)
I20260806 04:42:54.452247 16976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:42:54.456569 17050 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:42:54.458221 17050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Bootstrap starting.
I20260806 04:42:54.459012 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 16976
I20260806 04:42:54.459534 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:42471
--local_ip_for_outbound_sockets=127.31.166.129
--tserver_master_addrs=127.31.166.190:33009
--webserver_port=36533
--webserver_interface=127.31.166.129
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:42:54.481693 17050 log.cc:824] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:54.493067 17050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:42:54.493556 17050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Bootstrap complete.
I20260806 04:42:54.499177 17050 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } }
I20260806 04:42:54.499688 17050 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1997214492e435b9ac7975570c3e62c, State: Initialized, Role: FOLLOWER
I20260806 04:42:54.500236 17050 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } }
I20260806 04:42:54.500379 17050 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:42:54.500454 17050 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:42:54.500552 17050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:42:54.503197 17050 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } }
I20260806 04:42:54.503582 17050 leader_election.cc:302] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1997214492e435b9ac7975570c3e62c; no voters:
I20260806 04:42:54.503968 17050 leader_election.cc:288] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 04:42:54.504113 17054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 04:42:54.504570 17054 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 2 LEADER]: Becoming Leader. State: Replica: e1997214492e435b9ac7975570c3e62c, State: Running, Role: LEADER
I20260806 04:42:54.504909 17054 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } }
I20260806 04:42:54.505250 17050 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:42:54.506389 17056 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1997214492e435b9ac7975570c3e62c. Latest consensus state: current_term: 2 leader_uuid: "e1997214492e435b9ac7975570c3e62c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } } }
I20260806 04:42:54.506544 17056 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:54.506832 17055 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e1997214492e435b9ac7975570c3e62c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } } }
I20260806 04:42:54.506910 17055 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: This master's current role is: LEADER
I20260806 04:42:54.506937 17064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:42:54.511888 17064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:42:54.512759 17064 catalog_manager.cc:1259] Loaded cluster ID: 4534b88ffbd04685ac7c020d9ffda2b3
I20260806 04:42:54.512835 17064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:42:54.514700 17064 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:42:54.515381 17064 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Loaded TSK: 0
I20260806 04:42:54.530689 17064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Generated new TSK 1
I20260806 04:42:54.530975 17064 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:42:54.617889 17052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:54.618265 17052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:54.618332 17052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:54.624800 17052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:54.624909 17052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:54.625012 17052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:42:54.634172 17052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:42471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=36533
--tserver_master_addrs=127.31.166.190:33009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:54.635787 17052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:54.637226 17052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:54.648818 17083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:54.648886 17079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:54.648885 17080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:54.649382 17052 server_base.cc:1034] running on GCE node
I20260806 04:42:54.649870 17052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:54.650840 17052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:54.652061 17052 hybrid_clock.cc:648] HybridClock initialized: now 1785991374652019 us; error 38 us; skew 500 ppm
Aug 06 04:42:54 dist-test-slave-vkjp krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991374, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:54.655615 17052 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:42:54.656517 17052 webserver.cc:460] Webserver started at http://127.31.166.129:36533/ using document root <none> and password file <none>
I20260806 04:42:54.657135 17052 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:54.657248 17052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:54.661418 17052 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:42:54.663691 17090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:54.665040 17052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:42:54.665174 17052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "1183be6c6cc74149b98167e49fb300d8"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "86278051fd3f42b6989908daa9bda96b"
server_key_iv: "ea0835be23a2910fee03c6d9b33f4709"
server_key_version: "encryptionkey@0"
I20260806 04:42:54.665704 17052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:54.675978 17052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:54.679239 17052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:54.679437 17052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:54.680022 17052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:54.681147 17097 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 04:42:54.686760 17052 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 04:42:54.686841 17052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260806 04:42:54.686923 17052 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 04:42:54.688714 17052 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 04:42:54.688791 17052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 04:42:54.689092 17097 tablet_bootstrap.cc:492] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: Bootstrap starting.
I20260806 04:42:54.699654 17052 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:42471
I20260806 04:42:54.700595 17052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:42:54.701452 17208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:42471 every 8 connection(s)
I20260806 04:42:54.707706 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 17052
I20260806 04:42:54.708173 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:39161
--local_ip_for_outbound_sockets=127.31.166.130
--tserver_master_addrs=127.31.166.190:33009
--webserver_port=34321
--webserver_interface=127.31.166.130
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:42:54 dist-test-slave-vkjp krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991374, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:54.713413 17097 log.cc:824] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:54.719390 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:54.702709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:37835 (local address 127.31.166.190:33009)
0806 04:42:54.703835 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:54.703851 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:54.703915 (+ 64us) server_negotiation.cc:407] Connection header received
0806 04:42:54.704786 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:54.704796 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:54.705122 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:54.705427 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:54.706999 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:54.708499 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:54.710271 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:54.710983 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:54.711197 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:54.714086 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:54.714093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:54.714108 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:54.714141 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:54.715980 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:54.716538 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:54.716543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:54.716547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:54.716664 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:54.717028 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:54.717032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:54.717034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:54.717461 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:54.717624 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:54.718165 (+ 541us) server_negotiation.cc:299] Negotiation successful
0806 04:42:54.718427 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":104,"threads_started":1}
I20260806 04:42:54.719396 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:54.702747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:37835)
0806 04:42:54.703323 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:54.703376 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:54.704381 (+ 1005us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:54.705494 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:54.705502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:54.706222 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:54.706795 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:54.706812 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:54.708706 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:54.708712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:54.710136 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:54.710146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:54.711445 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:54.711449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:54.711655 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:54.712292 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:54.712309 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:54.713850 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:54.716117 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:54.716123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:54.716135 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:54.716409 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:54.716768 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:54.716771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:54.716773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:54.716931 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:54.717596 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:54.717602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:54.717854 (+ 252us) client_negotiation.cc:769] Sending connection context
0806 04:42:54.718051 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 04:42:54.718406 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":154,"threads_started":1}
I20260806 04:42:54.722101 17209 heartbeater.cc:347] Connected to a master server at 127.31.166.190:33009
I20260806 04:42:54.722352 17209 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:54.722851 17209 heartbeater.cc:511] Master 127.31.166.190:33009 requested a full tablet report, sending...
I20260806 04:42:54.725234 17011 ts_manager.cc:194] Registered new tserver with Master: 1183be6c6cc74149b98167e49fb300d8 (127.31.166.129:42471)
I20260806 04:42:54.728760 17011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:37835
I20260806 04:42:54.732128 17097 tablet_bootstrap.cc:492] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:42:54.732591 17097 tablet_bootstrap.cc:492] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: Bootstrap complete.
I20260806 04:42:54.734189 17097 ts_tablet_manager.cc:1397] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: Time spent bootstrapping tablet: real 0.045s user 0.024s sys 0.009s
I20260806 04:42:54.734627 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:54.728076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:42471 (local address 127.0.0.1:57352)
0806 04:42:54.729383 (+ 1307us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:54.729429 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:54.729564 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:54.730892 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:54.730898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:54.730921 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:54.731038 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:54.731045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:54.732114 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:54.732118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:54.733098 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:54.733108 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:54.734013 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:54.734016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:54.734172 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:54.734217 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 04:42:54.734335 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 04:42:54.734440 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1138,"thread_start_us":126,"threads_started":1}
I20260806 04:42:54.735239 17216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:54.728820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.31.166.129:42471)
0806 04:42:54.730034 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:54.730045 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:54.730068 (+ 23us) server_negotiation.cc:407] Connection header received
0806 04:42:54.730134 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:54.730139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:54.730599 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:54.730752 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:54.731199 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:54.731989 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:54.733221 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:54.733887 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:54.734016 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:54.734150 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:54.735002 (+ 852us) server_negotiation.cc:299] Negotiation successful
0806 04:42:54.735119 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1121,"thread_start_us":107,"threads_started":1}
I20260806 04:42:54.736145 17141 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f71c41e44e74fd9b495d2086c71365e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:34 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:57352
W20260806 04:42:54.738068 16994 catalog_manager.cc:4821] TS 1183be6c6cc74149b98167e49fb300d8 (127.31.166.129:42471): delete failed for tablet 6f71c41e44e74fd9b495d2086c71365e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f71c41e44e74fd9b495d2086c71365e already in progress: opening tablet
I20260806 04:42:54.737954 17097 raft_consensus.cc:348] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } } peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } }
I20260806 04:42:54.738534 17097 raft_consensus.cc:729] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1183be6c6cc74149b98167e49fb300d8, State: Initialized, Role: FOLLOWER
I20260806 04:42:54.739037 17097 consensus_queue.cc:260] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } } peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } }
I20260806 04:42:54.739715 17209 heartbeater.cc:503] Master 127.31.166.190:33009 was elected leader, sending a full tablet report...
I20260806 04:42:54.740155 17097 ts_tablet_manager.cc:1428] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260806 04:42:54.741322 17141 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f71c41e44e74fd9b495d2086c71365e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:34 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:57352
I20260806 04:42:54.741772 17217 tablet_replica.cc:330] stopping tablet replica
I20260806 04:42:54.742092 17217 raft_consensus.cc:2229] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:42:54.742332 17217 raft_consensus.cc:2258] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:42:54.743501 17217 ts_tablet_manager.cc:1905] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:42:54.746871 17217 ts_tablet_manager.cc:1918] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:42:54.746948 17217 log.cc:1196] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal/wals/6f71c41e44e74fd9b495d2086c71365e
I20260806 04:42:54.747251 17217 ts_tablet_manager.cc:1939] T 6f71c41e44e74fd9b495d2086c71365e P 1183be6c6cc74149b98167e49fb300d8: Deleting consensus metadata
I20260806 04:42:54.748400 16994 catalog_manager.cc:4841] TS 1183be6c6cc74149b98167e49fb300d8 (127.31.166.129:42471): tablet 6f71c41e44e74fd9b495d2086c71365e (table test-table [id=10c35abf297a4eaa83c2ff1ca51e3cfb]) successfully deleted
W20260806 04:42:54.838449 17213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:54.838723 17213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:54.838779 17213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:54.843150 17213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:54.843209 17213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:54.843266 17213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:42:54.848376 17213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:39161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=34321
--tserver_master_addrs=127.31.166.190:33009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:54.849366 17213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:54.850394 17213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:54.859575 17224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:54.859624 17225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:54.859565 17227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:54.859728 17213 server_base.cc:1034] running on GCE node
I20260806 04:42:54.860316 17213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:54.860939 17213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:54.862128 17213 hybrid_clock.cc:648] HybridClock initialized: now 1785991374862100 us; error 41 us; skew 500 ppm
Aug 06 04:42:54 dist-test-slave-vkjp krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991374, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:54.864977 17213 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:42:54.865826 17213 webserver.cc:460] Webserver started at http://127.31.166.130:34321/ using document root <none> and password file <none>
I20260806 04:42:54.866384 17213 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:54.866514 17213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:54.870656 17213 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:42:54.872799 17234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:54.874034 17213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:42:54.874167 17213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "195bf04a5e674522a9648b8213e85fb1"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "a649efed10885a1746402dfcd6b12cf4"
server_key_iv: "eacc6f3e8bed8c6298edca9cbd32a5a9"
server_key_version: "encryptionkey@0"
I20260806 04:42:54.874719 17213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:54.894300 17213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:54.897375 17213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:54.897609 17213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:54.898239 17213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:54.899204 17213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:54.899271 17213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:54.899343 17213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:54.899394 17213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:54.909206 17213 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:39161
I20260806 04:42:54.909237 17353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:39161 every 8 connection(s)
I20260806 04:42:54.910197 17213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:42:54.910730 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 17213
I20260806 04:42:54.911245 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:42257
--local_ip_for_outbound_sockets=127.31.166.131
--tserver_master_addrs=127.31.166.190:33009
--webserver_port=38093
--webserver_interface=127.31.166.131
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 04:42:54 dist-test-slave-vkjp krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991374, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:54.924909 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:54.912221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:36211 (local address 127.31.166.190:33009)
0806 04:42:54.912357 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:54.912361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:54.913679 (+ 1318us) server_negotiation.cc:407] Connection header received
0806 04:42:54.914560 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:54.914566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:54.914634 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:54.914737 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:54.915993 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:54.916927 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:54.918059 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:54.918520 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:54.918658 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:54.921089 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:54.921097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:54.921102 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:54.921134 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:54.922847 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:54.923393 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:54.923396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:54.923398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:54.923492 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:54.923803 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:54.923807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:54.923809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:54.924031 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:54.924127 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:54.924586 (+ 459us) server_negotiation.cc:299] Negotiation successful
0806 04:42:54.924725 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:42:54.925704 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:54.912734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:36211)
0806 04:42:54.913507 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:54.913559 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:54.914344 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:54.914888 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:54.914894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:54.915336 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:54.915826 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:54.915840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:54.917072 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:54.917075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:54.917929 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:54.917938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:54.918675 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:54.918677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:54.918820 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:54.919467 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:54.919484 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:54.920901 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:54.922967 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:54.922971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:54.922983 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:54.923301 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:54.923582 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:54.923584 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:54.923585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:54.923710 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:54.924125 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:54.924130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:54.924346 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 04:42:54.924505 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 04:42:54.924745 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":152,"threads_started":1}
I20260806 04:42:54.926882 17354 heartbeater.cc:347] Connected to a master server at 127.31.166.190:33009
I20260806 04:42:54.927153 17354 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:54.927698 17354 heartbeater.cc:511] Master 127.31.166.190:33009 requested a full tablet report, sending...
I20260806 04:42:54.928762 17011 ts_manager.cc:194] Registered new tserver with Master: 195bf04a5e674522a9648b8213e85fb1 (127.31.166.130:39161)
I20260806 04:42:54.929323 17011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:36211
W20260806 04:42:55.030066 17356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:42:55.030370 17356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:42:55.030452 17356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:42:55.034641 17356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:42:55.034720 17356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:42:55.034812 17356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:42:55.040325 17356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:42257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=38093
--tserver_master_addrs=127.31.166.190:33009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:42:55.041404 17356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:42:55.042439 17356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:42:55.052562 17364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:55.052594 17366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:42:55.052590 17362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:42:55.052796 17356 server_base.cc:1034] running on GCE node
I20260806 04:42:55.053316 17356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:42:55.053967 17356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:42:55.055189 17356 hybrid_clock.cc:648] HybridClock initialized: now 1785991375055152 us; error 43 us; skew 500 ppm
Aug 06 04:42:55 dist-test-slave-vkjp krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991375, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:42:55.058038 17356 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:42:55.058847 17356 webserver.cc:460] Webserver started at http://127.31.166.131:38093/ using document root <none> and password file <none>
I20260806 04:42:55.059439 17356 fs_manager.cc:359] Metadata directory not provided
I20260806 04:42:55.059517 17356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:42:55.063802 17356 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:42:55.065873 17373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:55.067243 17356 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:42:55.067353 17356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "edee860e107746ac9fb1601036495d9c"
format_stamp: "Formatted at 2026-08-06 04:42:34 on dist-test-slave-vkjp"
server_key: "8460bec3e85e87bb6b80256a9216a66a"
server_key_iv: "ea2672b1f80678242b9410f68727eed9"
server_key_version: "encryptionkey@0"
I20260806 04:42:55.067894 17356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:42:55.106724 17356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:42:55.109884 17356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:42:55.110054 17356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:42:55.110608 17356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:42:55.111485 17356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:42:55.111532 17356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:55.111575 17356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:42:55.111598 17356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:42:55.120498 17356 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:42257
I20260806 04:42:55.120527 17491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:42257 every 8 connection(s)
I20260806 04:42:55.121517 17356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:42:55.129037 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 17356
Aug 06 04:42:55 dist-test-slave-vkjp krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991375, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:42:55.135639 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.123349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:58459 (local address 127.31.166.190:33009)
0806 04:42:55.123523 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:55.123528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:55.124630 (+ 1102us) server_negotiation.cc:407] Connection header received
0806 04:42:55.125454 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:55.125456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:55.125501 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:55.125576 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:55.126746 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.127290 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.128409 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.128627 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.128703 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:55.131398 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:55.131402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:55.131406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:55.131442 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:55.133038 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:55.133603 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:55.133607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:55.133609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:55.133701 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:55.134312 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:55.134316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:55.134318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:55.134507 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:55.134645 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:55.135277 (+ 632us) server_negotiation.cc:299] Negotiation successful
0806 04:42:55.135420 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 04:42:55.136516 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.123800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:58459)
0806 04:42:55.124479 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:55.124521 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:55.125262 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:55.125735 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:55.125741 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:55.126098 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:42:55.126617 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:55.126630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.127402 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.127405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:55.128305 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:55.128314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.128786 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.128790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:55.128934 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:55.129619 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:42:55.129640 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:42:55.131228 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:42:55.133151 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:55.133155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:55.133166 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:55.133520 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:55.134086 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:42:55.134089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:42:55.134090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:42:55.134215 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:42:55.134649 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:42:55.134656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:42:55.134992 (+ 336us) client_negotiation.cc:769] Sending connection context
0806 04:42:55.135239 (+ 247us) client_negotiation.cc:240] Negotiation successful
0806 04:42:55.135539 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":125,"threads_started":1}
I20260806 04:42:55.137699 17492 heartbeater.cc:347] Connected to a master server at 127.31.166.190:33009
I20260806 04:42:55.137971 17492 heartbeater.cc:464] Registering TS with master...
I20260806 04:42:55.138563 17492 heartbeater.cc:511] Master 127.31.166.190:33009 requested a full tablet report, sending...
I20260806 04:42:55.138850 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.129773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.31.166.190:33009)
0806 04:42:55.130034 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:55.130039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:55.130238 (+ 199us) server_negotiation.cc:407] Connection header received
0806 04:42:55.130397 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:55.130400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:55.130460 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:55.130555 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:55.131408 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.132084 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.133354 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.133701 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.133805 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:55.135381 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:55.135386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:55.135389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:55.135411 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:55.136954 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:55.137404 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:55.137407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:55.137408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:55.137485 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:55.137773 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:55.137778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:55.137783 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:55.138001 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:55.138214 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:55.138554 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 04:42:55.138676 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":111,"threads_started":1}
I20260806 04:42:55.139657 17011 ts_manager.cc:194] Registered new tserver with Master: edee860e107746ac9fb1601036495d9c (127.31.166.131:42257)
I20260806 04:42:55.140339 17011 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:58459
I20260806 04:42:55.140820 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:42:55.145247 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.141483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51670 (local address 127.31.166.190:33009)
0806 04:42:55.141606 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:55.141613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:55.142516 (+ 903us) server_negotiation.cc:407] Connection header received
0806 04:42:55.142628 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:55.142630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:55.142709 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:55.142785 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:42:55.142979 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.143505 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.144291 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.144509 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.144581 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:55.144823 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:42:55.144933 (+ 110us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 04:42:55.145138 (+ 205us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51670: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 04:42:55.145316 17495 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51670: token expired
W20260806 04:42:55.145655 16933 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.166.190:33009)
I20260806 04:42:55.152554 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.145980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.31.166.190:33009)
0806 04:42:55.146103 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:55.146106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:55.146117 (+ 11us) server_negotiation.cc:407] Connection header received
0806 04:42:55.146202 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:55.146204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:55.146234 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:55.146292 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:42:55.146841 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.147349 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.148225 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.148413 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.148495 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:55.149705 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:42:55.149708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:42:55.149710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:42:55.149730 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:42:55.151221 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:55.151578 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:55.151581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:55.151582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:55.151650 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:42:55.151867 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:42:55.151869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:42:55.151870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:42:55.152048 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:42:55.152146 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:55.152309 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 04:42:55.152390 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:42:55.155346 17011 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:42:55.158857 17011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:42:55.167572 17141 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9f5f1594914c9aa5369d0d17d3b972 (DEFAULT_TABLE table=test-table [id=559375b6ed83499c946078b17bc090be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:55.168283 17141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9f5f1594914c9aa5369d0d17d3b972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:55.171267 17097 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8: Bootstrap starting.
I20260806 04:42:55.171645 17215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.166318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:39161 (local address 127.0.0.1:33136)
0806 04:42:55.166501 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:55.166518 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:55.166596 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:55.167485 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:55.167488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:55.167507 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:55.167621 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:55.167628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.168974 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.168979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:55.170102 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:55.170113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.171236 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.171241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:55.171351 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:55.171378 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 04:42:55.171433 (+ 55us) client_negotiation.cc:240] Negotiation successful
0806 04:42:55.171512 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260806 04:42:55.172228 17097 tablet_bootstrap.cc:654] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:55.172413 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.166493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.31.166.130:39161)
0806 04:42:55.166869 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:55.166884 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:55.166908 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:42:55.167004 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:55.167008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:55.167284 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:55.167508 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:55.167744 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.168853 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.170219 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.171072 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.171981 (+ 909us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:55.172123 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:55.172177 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 04:42:55.172270 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":132,"threads_started":1}
I20260806 04:42:55.173166 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.166605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:42257 (local address 127.0.0.1:43820)
0806 04:42:55.167286 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:55.167300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:55.167406 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:55.168479 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:55.168482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:55.168499 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:55.168581 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:55.168586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.169812 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.169814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:55.170872 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:55.170885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.172754 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.172758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:55.172863 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:55.172882 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:42:55.172971 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 04:42:55.173043 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":108,"threads_started":1}
I20260806 04:42:55.173233 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.166848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.31.166.131:42257)
0806 04:42:55.167225 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:55.167236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:55.167397 (+ 161us) server_negotiation.cc:407] Connection header received
0806 04:42:55.167823 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:55.167829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:55.168009 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:55.168291 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:55.168796 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.169707 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.171425 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.172393 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.172531 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:55.172686 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:55.173004 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 04:42:55.173096 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20260806 04:42:55.173962 17097 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8: No bootstrap required, opened a new log
I20260806 04:42:55.174067 17097 ts_tablet_manager.cc:1397] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260806 04:42:55.174485 17097 raft_consensus.cc:348] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:42:55.174609 17097 raft_consensus.cc:374] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:55.174649 17097 raft_consensus.cc:729] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1183be6c6cc74149b98167e49fb300d8, State: Initialized, Role: FOLLOWER
I20260806 04:42:55.174774 17097 consensus_queue.cc:260] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:42:55.175029 17097 ts_tablet_manager.cc:1428] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260806 04:42:55.175714 17285 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9f5f1594914c9aa5369d0d17d3b972 (DEFAULT_TABLE table=test-table [id=559375b6ed83499c946078b17bc090be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:55.176100 17422 tablet_service.cc:1467] Processing CreateTablet for tablet 1d9f5f1594914c9aa5369d0d17d3b972 (DEFAULT_TABLE table=test-table [id=559375b6ed83499c946078b17bc090be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:42:55.176786 17285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9f5f1594914c9aa5369d0d17d3b972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:55.177112 17422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9f5f1594914c9aa5369d0d17d3b972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:42:55.183949 17505 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: Bootstrap starting.
I20260806 04:42:55.184180 17506 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Bootstrap starting.
I20260806 04:42:55.186307 17505 tablet_bootstrap.cc:654] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:55.186306 17506 tablet_bootstrap.cc:654] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Neither blocks nor log segments found. Creating new log.
I20260806 04:42:55.187031 17506 log.cc:824] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:55.187031 17505 log.cc:824] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: Log is configured to *not* fsync() on all Append() calls
I20260806 04:42:55.189361 17506 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: No bootstrap required, opened a new log
I20260806 04:42:55.189401 17505 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: No bootstrap required, opened a new log
I20260806 04:42:55.189613 17506 ts_tablet_manager.cc:1397] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:42:55.189615 17505 ts_tablet_manager.cc:1397] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:42:55.192631 17505 raft_consensus.cc:348] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:42:55.192631 17506 raft_consensus.cc:348] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:42:55.192860 17505 raft_consensus.cc:374] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:55.192889 17506 raft_consensus.cc:374] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:42:55.192926 17505 raft_consensus.cc:729] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195bf04a5e674522a9648b8213e85fb1, State: Initialized, Role: FOLLOWER
I20260806 04:42:55.192956 17506 raft_consensus.cc:729] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edee860e107746ac9fb1601036495d9c, State: Initialized, Role: FOLLOWER
I20260806 04:42:55.193378 17505 consensus_queue.cc:260] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:42:55.193378 17506 consensus_queue.cc:260] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:42:55.194226 17492 heartbeater.cc:503] Master 127.31.166.190:33009 was elected leader, sending a full tablet report...
I20260806 04:42:55.194424 17505 ts_tablet_manager.cc:1428] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 04:42:55.195240 17354 heartbeater.cc:503] Master 127.31.166.190:33009 was elected leader, sending a full tablet report...
I20260806 04:42:55.195472 17506 ts_tablet_manager.cc:1428] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
W20260806 04:42:55.202500 17210 tablet.cc:2369] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:42:55.322893 17511 raft_consensus.cc:482] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:42:55.323153 17511 raft_consensus.cc:504] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:42:55.324373 17511 leader_election.cc:288] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edee860e107746ac9fb1601036495d9c (127.31.166.131:42257), 1183be6c6cc74149b98167e49fb300d8 (127.31.166.129:42471)
I20260806 04:42:55.328502 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.324620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:42257 (local address 127.31.166.130:42769)
0806 04:42:55.324792 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:55.324811 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:55.324915 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:55.325306 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:55.325309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:55.325328 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:55.325513 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:55.325521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.326397 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.326400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:55.327320 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:55.327329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.328016 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.328021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:55.328128 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:55.328157 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 04:42:55.328227 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 04:42:55.328309 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"spinlock_wait_cycles":20352}
I20260806 04:42:55.328508 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.324740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:42769 (local address 127.31.166.131:42257)
0806 04:42:55.324884 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:55.324889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:55.324905 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:42:55.325054 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:55.325057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:55.325106 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:55.325177 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:55.325668 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.326282 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.327455 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.327889 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.327987 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:55.328069 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:55.328274 (+ 205us) server_negotiation.cc:299] Negotiation successful
0806 04:42:55.328353 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:42:55.328524 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.324908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:42471 (local address 127.31.166.130:35877)
0806 04:42:55.325279 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:55.325291 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:42:55.325387 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:42:55.325668 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:42:55.325670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:42:55.325687 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:42:55.325801 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:55.325807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.326569 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.326572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:55.327408 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:42:55.327420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.328092 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.328097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:42:55.328184 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:55.328201 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 04:42:55.328300 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 04:42:55.328363 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"spinlock_wait_cycles":1280,"thread_start_us":171,"threads_started":1}
I20260806 04:42:55.328547 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.325049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:35877 (local address 127.31.166.129:42471)
0806 04:42:55.325327 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:55.325332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:55.325346 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:42:55.325498 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:55.325502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:55.325555 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:55.325643 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:42:55.325911 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.326452 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.327536 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.327923 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.328081 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:55.328135 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:55.328331 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 04:42:55.328413 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
I20260806 04:42:55.329236 17161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9f5f1594914c9aa5369d0d17d3b972" candidate_uuid: "195bf04a5e674522a9648b8213e85fb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1183be6c6cc74149b98167e49fb300d8" is_pre_election: true
I20260806 04:42:55.329294 17443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9f5f1594914c9aa5369d0d17d3b972" candidate_uuid: "195bf04a5e674522a9648b8213e85fb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edee860e107746ac9fb1601036495d9c" is_pre_election: true
I20260806 04:42:55.329586 17443 raft_consensus.cc:2454] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195bf04a5e674522a9648b8213e85fb1 in term 0.
I20260806 04:42:55.329569 17161 raft_consensus.cc:2454] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195bf04a5e674522a9648b8213e85fb1 in term 0.
I20260806 04:42:55.330026 17235 leader_election.cc:302] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1183be6c6cc74149b98167e49fb300d8, 195bf04a5e674522a9648b8213e85fb1; no voters:
I20260806 04:42:55.330281 17511 raft_consensus.cc:2781] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:42:55.330369 17511 raft_consensus.cc:482] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:42:55.330415 17511 raft_consensus.cc:3037] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:55.331502 17511 raft_consensus.cc:504] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:42:55.331931 17511 leader_election.cc:288] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 1 election: Requested vote from peers edee860e107746ac9fb1601036495d9c (127.31.166.131:42257), 1183be6c6cc74149b98167e49fb300d8 (127.31.166.129:42471)
I20260806 04:42:55.332249 17443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9f5f1594914c9aa5369d0d17d3b972" candidate_uuid: "195bf04a5e674522a9648b8213e85fb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edee860e107746ac9fb1601036495d9c"
I20260806 04:42:55.332314 17161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9f5f1594914c9aa5369d0d17d3b972" candidate_uuid: "195bf04a5e674522a9648b8213e85fb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1183be6c6cc74149b98167e49fb300d8"
I20260806 04:42:55.332352 17443 raft_consensus.cc:3037] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:55.332472 17161 raft_consensus.cc:3037] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:42:55.333487 17443 raft_consensus.cc:2454] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195bf04a5e674522a9648b8213e85fb1 in term 1.
I20260806 04:42:55.333565 17161 raft_consensus.cc:2454] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195bf04a5e674522a9648b8213e85fb1 in term 1.
I20260806 04:42:55.333813 17235 leader_election.cc:302] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 195bf04a5e674522a9648b8213e85fb1, edee860e107746ac9fb1601036495d9c; no voters:
I20260806 04:42:55.334024 17511 raft_consensus.cc:2781] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:42:55.334295 17511 raft_consensus.cc:686] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 1 LEADER]: Becoming Leader. State: Replica: 195bf04a5e674522a9648b8213e85fb1, State: Running, Role: LEADER
I20260806 04:42:55.334661 17511 consensus_queue.cc:237] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:42:55.337829 17011 catalog_manager.cc:5495] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 195bf04a5e674522a9648b8213e85fb1 (127.31.166.130). New cstate: current_term: 1 leader_uuid: "195bf04a5e674522a9648b8213e85fb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } health_report { overall_health: UNKNOWN } } }
I20260806 04:42:55.366808 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.363593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33150 (local address 127.31.166.130:39161)
0806 04:42:55.363711 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 04:42:55.363715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:42:55.363729 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:42:55.363843 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:42:55.363845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:42:55.363901 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:42:55.364009 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:42:55.364220 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.364702 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.365478 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:42:55.365679 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:42:55.365754 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:42:55.366030 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:42:55.366426 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:42:55.366519 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 04:42:55.366582 (+ 63us) server_negotiation.cc:299] Negotiation successful
0806 04:42:55.366630 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260806 04:42:55.372442 17493 tablet.cc:2369] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:42:55.372766 17161 raft_consensus.cc:1261] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Refusing update from remote peer 195bf04a5e674522a9648b8213e85fb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:42:55.372822 17443 raft_consensus.cc:1261] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c [term 1 FOLLOWER]: Refusing update from remote peer 195bf04a5e674522a9648b8213e85fb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:42:55.373421 17511 consensus_queue.cc:1035] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:42:55.373576 17514 consensus_queue.cc:1035] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:42:55.381279 17519 mvcc.cc:205] Tried to move back new op lower bound from 7315420673519198208 to 7315420673373536256. Current Snapshot: MvccSnapshot[applied={T|T < 7315420673519198208}]
I20260806 04:42:55.387849 17011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:42:55.388046 17011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:42:55.390477 17011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Sending DeleteTablet for 3 replicas of tablet 1d9f5f1594914c9aa5369d0d17d3b972
I20260806 04:42:55.391263 17141 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d9f5f1594914c9aa5369d0d17d3b972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:55 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:57352
I20260806 04:42:55.391348 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 17052
I20260806 04:42:55.393323 17422 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d9f5f1594914c9aa5369d0d17d3b972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:55 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:43820
I20260806 04:42:55.395150 17285 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d9f5f1594914c9aa5369d0d17d3b972 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:42:55 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:33136
I20260806 04:42:55.395527 17527 tablet_replica.cc:330] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: stopping tablet replica
I20260806 04:42:55.395797 17527 raft_consensus.cc:2229] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:42:55.396051 17527 raft_consensus.cc:2258] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:42:55.396387 17528 tablet_replica.cc:330] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: stopping tablet replica
I20260806 04:42:55.396672 17528 raft_consensus.cc:2229] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:42:55.397054 17528 raft_consensus.cc:2258] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:42:55.397519 17527 ts_tablet_manager.cc:1905] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:42:55.400162 17527 ts_tablet_manager.cc:1918] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:42:55.400276 17527 log.cc:1196] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/1d9f5f1594914c9aa5369d0d17d3b972
I20260806 04:42:55.400647 17527 ts_tablet_manager.cc:1939] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Deleting consensus metadata
I20260806 04:42:55.401072 17528 ts_tablet_manager.cc:1905] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 04:42:55.401643 16994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:42471 (error 108)
I20260806 04:42:55.402295 16994 catalog_manager.cc:4841] TS edee860e107746ac9fb1601036495d9c (127.31.166.131:42257): tablet 1d9f5f1594914c9aa5369d0d17d3b972 (table test-table [id=559375b6ed83499c946078b17bc090be]) successfully deleted
W20260806 04:42:55.402962 17502 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:42:55.402637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:42471 (local address 127.0.0.1:57368)
0806 04:42:55.402807 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:55.402888 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:42471: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 04:42:55.403240 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 17213
W20260806 04:42:55.403178 16994 catalog_manager.cc:4584] TS 1183be6c6cc74149b98167e49fb300d8 (127.31.166.129:42471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d9f5f1594914c9aa5369d0d17d3b972: Network error: Client connection negotiation failed: client connection to 127.31.166.129:42471: connect: Connection reset by peer (error 104)
I20260806 04:42:55.411142 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 17356
I20260806 04:42:55.411350 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:42:55.410997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:39161 (local address 127.0.0.1:33164)
0806 04:42:55.411172 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 04:42:55.411243 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.130:39161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 04:42:55.417826 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 16976
I20260806 04:43:15.425581 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:33009
--webserver_interface=127.31.166.190
--webserver_port=38217
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:33009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:15.536791 17530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:15.537066 17530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:15.537107 17530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:15.541127 17530 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 04:43:15.541189 17530 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 04:43:15.541205 17530 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 04:43:15.541219 17530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:15.541236 17530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:15.541255 17530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:15.541271 17530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:15.541299 17530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:15.546638 17530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:33009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:33009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=38217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:15.547729 17530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:15.548688 17530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:15.557855 17536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:15.557850 17535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:15.557928 17530 server_base.cc:1034] running on GCE node
W20260806 04:43:15.557849 17538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:15.558694 17530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:15.559937 17530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:15.561146 17530 hybrid_clock.cc:648] HybridClock initialized: now 1785991395561133 us; error 33 us; skew 500 ppm
Aug 06 04:43:15 dist-test-slave-vkjp krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991395, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:15.569514 17530 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:43:15.570571 17530 webserver.cc:460] Webserver started at http://127.31.166.190:38217/ using document root <none> and password file <none>
I20260806 04:43:15.571179 17530 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:15.571285 17530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:15.575503 17530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:43:15.578019 17545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:15.579473 17530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:43:15.579634 17530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "e1997214492e435b9ac7975570c3e62c"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "35ac87b18bd3733756eae49578ab1f20"
server_key_iv: "d3b2453c06b4db0308f151905ec4b305"
server_key_version: "encryptionkey@0"
I20260806 04:43:15.580232 17530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:15.599323 17530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:15.602852 17530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:15.603060 17530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:15.610342 17530 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:33009
I20260806 04:43:15.610627 17599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:33009 every 8 connection(s)
I20260806 04:43:15.611527 17530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:15.612748 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 17530
I20260806 04:43:15.613188 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:42471
--local_ip_for_outbound_sockets=127.31.166.129
--tserver_master_addrs=127.31.166.190:33009
--webserver_port=36533
--webserver_interface=127.31.166.129
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:43:15.616611 17600 sys_catalog.cc:263] Verifying existing consensus state
I20260806 04:43:15.618425 17600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Bootstrap starting.
I20260806 04:43:15.637742 17600 log.cc:824] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:15.650141 17600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:43:15.650625 17600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Bootstrap complete.
I20260806 04:43:15.655498 17600 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } }
I20260806 04:43:15.655974 17600 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1997214492e435b9ac7975570c3e62c, State: Initialized, Role: FOLLOWER
I20260806 04:43:15.656617 17600 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } }
I20260806 04:43:15.656767 17600 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:15.656865 17600 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:15.657006 17600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:43:15.659341 17600 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } }
I20260806 04:43:15.659729 17600 leader_election.cc:302] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1997214492e435b9ac7975570c3e62c; no voters:
I20260806 04:43:15.660360 17600 leader_election.cc:288] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 04:43:15.660424 17604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:43:15.660635 17604 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [term 3 LEADER]: Becoming Leader. State: Replica: e1997214492e435b9ac7975570c3e62c, State: Running, Role: LEADER
I20260806 04:43:15.661041 17604 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } }
I20260806 04:43:15.661746 17600 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:15.662676 17605 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1997214492e435b9ac7975570c3e62c. Latest consensus state: current_term: 3 leader_uuid: "e1997214492e435b9ac7975570c3e62c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } } }
I20260806 04:43:15.662827 17605 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:15.663125 17606 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e1997214492e435b9ac7975570c3e62c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1997214492e435b9ac7975570c3e62c" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33009 } } }
I20260806 04:43:15.663210 17613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:15.663219 17606 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:15.669086 17613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:15.669909 17613 catalog_manager.cc:1259] Loaded cluster ID: 4534b88ffbd04685ac7c020d9ffda2b3
I20260806 04:43:15.669991 17613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:15.671880 17613 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:15.672513 17613 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Loaded TSK: 0
I20260806 04:43:15.672600 17613 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e1997214492e435b9ac7975570c3e62c: Loaded TSK: 1
F20260806 04:43:15.673207 17613 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f41246f79ad google::LogMessage::Fail() at ??:0
@ 0x7f41246fa88a google::LogMessage::SendToLog() at ??:0
@ 0x7f41246f74de google::LogMessage::Flush() at ??:0
@ 0x7f41246f8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f41273bb18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f41273b3f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f412f05c7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f412f03699c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f412f037178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f412f078df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f412f09782b std::function<>::operator()() at ??:0
@ 0x7f412f036b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f412f037f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f412f034579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f412f078640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f41269493d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4126949c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f412694b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f412693a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4127f7e6db start_thread at ??:0
@ 0x7f41231f561f clone at ??:0
W20260806 04:43:15.770673 17602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:15.771147 17602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:15.771247 17602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:15.777994 17602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:15.778097 17602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:15.778208 17602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:43:15.786403 17602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:42471
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=36533
--tserver_master_addrs=127.31.166.190:33009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:15.787971 17602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:15.789387 17602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:15.802047 17650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:15.802042 17651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:15.802078 17653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:15.802716 17602 server_base.cc:1034] running on GCE node
I20260806 04:43:15.803184 17602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:15.803920 17602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:15.805097 17602 hybrid_clock.cc:648] HybridClock initialized: now 1785991395805073 us; error 42 us; skew 500 ppm
Aug 06 04:43:15 dist-test-slave-vkjp krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991395, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:15.807927 17602 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:43:15.808830 17602 webserver.cc:460] Webserver started at http://127.31.166.129:36533/ using document root <none> and password file <none>
I20260806 04:43:15.809399 17602 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:15.809460 17602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:15.813566 17602 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:43:15.815853 17661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:15.817196 17602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 04:43:15.817325 17602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "1183be6c6cc74149b98167e49fb300d8"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "86278051fd3f42b6989908daa9bda96b"
server_key_iv: "ea0835be23a2910fee03c6d9b33f4709"
server_key_version: "encryptionkey@0"
I20260806 04:43:15.817896 17602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:15.847241 17602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:15.850556 17602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:15.850764 17602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:15.851444 17602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:15.852571 17668 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 04:43:15.857758 17602 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 04:43:15.857822 17602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260806 04:43:15.857889 17602 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 04:43:15.859961 17602 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 04:43:15.860075 17602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260806 04:43:15.860317 17668 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8: Bootstrap starting.
I20260806 04:43:15.870690 17602 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:42471
I20260806 04:43:15.870934 17783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:42471 every 8 connection(s)
I20260806 04:43:15.871757 17602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:43:15.881222 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 17602
I20260806 04:43:15.881731 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:39161
--local_ip_for_outbound_sockets=127.31.166.130
--tserver_master_addrs=127.31.166.190:33009
--webserver_port=34321
--webserver_interface=127.31.166.130
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:43:15.882411 17668 log.cc:824] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:15.889521 17668 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:43:15.889917 17668 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8: Bootstrap complete.
I20260806 04:43:15.891031 17668 ts_tablet_manager.cc:1397] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.005s
I20260806 04:43:15.893383 17668 raft_consensus.cc:348] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:43:15.893837 17668 raft_consensus.cc:729] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1183be6c6cc74149b98167e49fb300d8, State: Initialized, Role: FOLLOWER
I20260806 04:43:15.894376 17668 consensus_queue.cc:260] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:43:15.895345 17668 ts_tablet_manager.cc:1428] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20260806 04:43:15.941277 17786 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:43:15.873681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:56663)
0806 04:43:15.874168 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:15.874220 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:15.875040 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:15.940291 (+ 65251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":148,"threads_started":1}
W20260806 04:43:15.941998 17784 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:33009 (0 consecutive failures): Network error: Failed to ping master at 127.31.166.190:33009: Client connection negotiation failed: client connection to 127.31.166.190:33009: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 04:43:15.942900 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:15.942556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:38973)
0806 04:43:15.942725 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:15.942820 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 04:43:15.943638 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:15.943411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:52073)
0806 04:43:15.943525 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:15.943580 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 04:43:15.943836 17784 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 04:43:16.013332 17788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:16.013690 17788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:16.013816 17788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:16.018249 17788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:16.018363 17788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:16.018481 17788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:43:16.024113 17788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:39161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=34321
--tserver_master_addrs=127.31.166.190:33009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:16.025220 17788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:16.026257 17788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:16.035997 17800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:16.036135 17788 server_base.cc:1034] running on GCE node
W20260806 04:43:16.036034 17798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:16.036069 17797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:16.036698 17788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:16.037341 17788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:16.038529 17788 hybrid_clock.cc:648] HybridClock initialized: now 1785991396038509 us; error 36 us; skew 500 ppm
Aug 06 04:43:16 dist-test-slave-vkjp krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991396, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:16.041348 17788 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:43:16.042181 17788 webserver.cc:460] Webserver started at http://127.31.166.130:34321/ using document root <none> and password file <none>
I20260806 04:43:16.042695 17788 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:16.042753 17788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:16.046470 17788 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:43:16.048682 17808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:16.049937 17788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:43:16.050077 17788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "195bf04a5e674522a9648b8213e85fb1"
format_stamp: "Formatted at 2026-08-06 04:42:33 on dist-test-slave-vkjp"
server_key: "a649efed10885a1746402dfcd6b12cf4"
server_key_iv: "eacc6f3e8bed8c6298edca9cbd32a5a9"
server_key_version: "encryptionkey@0"
I20260806 04:43:16.050616 17788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:16.066325 17788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:16.069687 17788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:16.069900 17788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:16.070508 17788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:16.071630 17815 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260806 04:43:16.075721 17788 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 04:43:16.075793 17788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260806 04:43:16.075871 17788 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 04:43:16.077504 17788 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 04:43:16.077569 17788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260806 04:43:16.077845 17815 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: Bootstrap starting.
I20260806 04:43:16.083220 17815 log.cc:824] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:16.089682 17788 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:39161
I20260806 04:43:16.089768 17929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:39161 every 8 connection(s)
I20260806 04:43:16.090651 17788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:43:16.091324 17815 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:43:16.091949 17815 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: Bootstrap complete.
I20260806 04:43:16.092293 17815 ts_tablet_manager.cc:1397] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260806 04:43:16.094311 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:16.092794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:53475)
0806 04:43:16.093271 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:16.093407 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":109,"threads_started":1}
W20260806 04:43:16.094767 17930 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:33009 (0 consecutive failures): Network error: Failed to ping master at 127.31.166.190:33009: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
I20260806 04:43:16.095433 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:16.095201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:56753)
0806 04:43:16.095316 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:16.095370 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260806 04:43:16.096086 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:16.095899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:45609)
0806 04:43:16.095998 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:16.096038 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 04:43:16.096318 17930 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 04:43:16.095919 17815 raft_consensus.cc:348] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:43:16.096457 17815 raft_consensus.cc:729] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195bf04a5e674522a9648b8213e85fb1, State: Initialized, Role: FOLLOWER
I20260806 04:43:16.096897 17815 consensus_queue.cc:260] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:43:16.097811 17815 ts_tablet_manager.cc:1428] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 04:43:16.097883 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:16.097686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:42223)
0806 04:43:16.097783 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:16.097835 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 04:43:16.098538 17932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:16.098348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:49237)
0806 04:43:16.098452 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:16.098494 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 04:43:16.100013 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 17788
I20260806 04:43:16.100467 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:42257
--local_ip_for_outbound_sockets=127.31.166.131
--tserver_master_addrs=127.31.166.190:33009
--webserver_port=38093
--webserver_interface=127.31.166.131
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:33059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:16.211460 17934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:16.211714 17934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:16.211756 17934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:16.215863 17934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:16.215947 17934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:16.216038 17934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:43:16.221194 17934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:33059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:42257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=38093
--tserver_master_addrs=127.31.166.190:33009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:16.222324 17934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:16.223423 17934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:16.233094 17939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:16.233094 17942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:16.233094 17940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:16.233798 17934 server_base.cc:1034] running on GCE node
I20260806 04:43:16.234192 17934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:16.234809 17934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:16.235976 17934 hybrid_clock.cc:648] HybridClock initialized: now 1785991396235946 us; error 40 us; skew 500 ppm
Aug 06 04:43:16 dist-test-slave-vkjp krb5kdc[16396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991396, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:16.238885 17934 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:43:16.239933 17934 webserver.cc:460] Webserver started at http://127.31.166.131:38093/ using document root <none> and password file <none>
I20260806 04:43:16.240496 17934 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:16.240561 17934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:16.244467 17934 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:43:16.246527 17949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:16.247862 17934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:43:16.247977 17934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "edee860e107746ac9fb1601036495d9c"
format_stamp: "Formatted at 2026-08-06 04:42:34 on dist-test-slave-vkjp"
server_key: "8460bec3e85e87bb6b80256a9216a66a"
server_key_iv: "ea2672b1f80678242b9410f68727eed9"
server_key_version: "encryptionkey@0"
I20260806 04:43:16.248521 17934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:16.263244 17934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:16.266685 17934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:16.266935 17934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:16.267639 17934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:16.268533 17934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:16.268584 17934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:16.268656 17934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:16.268699 17934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:16.278932 17934 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:42257
I20260806 04:43:16.278965 18067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:42257 every 8 connection(s)
I20260806 04:43:16.280017 17934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:43:16.283674 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:16.282152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:58543)
0806 04:43:16.282819 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:16.282925 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":126,"threads_started":1}
W20260806 04:43:16.284081 18068 heartbeater.cc:644] Failed to heartbeat to 127.31.166.190:33009 (0 consecutive failures): Network error: Failed to ping master at 127.31.166.190:33009: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
I20260806 04:43:16.284636 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:16.284433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:37551)
0806 04:43:16.284536 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:16.284583 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260806 04:43:16.285179 18071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:16.285005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:42291)
0806 04:43:16.285096 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:16.285136 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260806 04:43:16.285385 18068 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 04:43:16.287205 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 17934
I20260806 04:43:16.945050 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:16.944549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:43029)
0806 04:43:16.944864 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:16.944929 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":89,"threads_started":1}
I20260806 04:43:17.099812 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:17.099339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:43621)
0806 04:43:17.099611 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:17.099699 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":73,"threads_started":1}
I20260806 04:43:17.286404 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:17.285958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:34971)
0806 04:43:17.286208 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:17.286264 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260806 04:43:17.526103 18077 raft_consensus.cc:482] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:43:17.526345 18077 raft_consensus.cc:504] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:43:17.527426 18078 raft_consensus.cc:482] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:43:17.527608 18078 raft_consensus.cc:504] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:43:17.527848 18077 leader_election.cc:288] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers edee860e107746ac9fb1601036495d9c (127.31.166.131:42257), 1183be6c6cc74149b98167e49fb300d8 (127.31.166.129:42471)
I20260806 04:43:17.528812 18078 leader_election.cc:288] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 195bf04a5e674522a9648b8213e85fb1 (127.31.166.130:39161), edee860e107746ac9fb1601036495d9c (127.31.166.131:42257)
Aug 06 04:43:17 dist-test-slave-vkjp krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991395, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.130@KRBTEST.COM
Aug 06 04:43:17 dist-test-slave-vkjp krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991396, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.129@KRBTEST.COM
Aug 06 04:43:17 dist-test-slave-vkjp krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991396, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.131@KRBTEST.COM
Aug 06 04:43:17 dist-test-slave-vkjp krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991395, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.131@KRBTEST.COM
I20260806 04:43:17.544517 18079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:17.528514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:42471 (local address 127.31.166.130:53657)
0806 04:43:17.529049 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:17.529068 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:17.529191 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:17.530289 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:17.530298 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:17.530808 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:17.531391 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:17.531412 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.533109 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.533113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:17.534286 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:17.534300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.535703 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.535819 (+ 116us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:17.535996 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:17.536635 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:43:17.536655 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:17.539457 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:17.541968 (+ 2511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:17.541979 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:17.541989 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:17.542310 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:17.542692 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:17.542696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:17.542698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:17.542852 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:17.543702 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:17.543707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:17.543829 (+ 122us) client_negotiation.cc:769] Sending connection context
0806 04:43:17.544005 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 04:43:17.544301 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":5,"thread_start_us":80,"threads_started":1}
I20260806 04:43:17.544517 18084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:17.529627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:39161 (local address 127.31.166.129:48075)
0806 04:43:17.529937 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:17.529952 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:17.530045 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:17.530660 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:17.530666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:17.531170 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:17.531784 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:17.531803 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.533410 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.533414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:17.534534 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:17.534542 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.535557 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.535560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:17.535756 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:17.536435 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:43:17.536477 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:17.538656 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:17.541985 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:17.541993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:17.542002 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:17.542248 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:17.542692 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:17.542696 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:17.542698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:17.542852 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:17.543649 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:17.543656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:17.543816 (+ 160us) client_negotiation.cc:769] Sending connection context
0806 04:43:17.543990 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 04:43:17.544299 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":71,"spinlock_wait_cycles":1920,"thread_start_us":58,"threads_started":1}
I20260806 04:43:17.545537 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:17.529810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:48075 (local address 127.31.166.130:39161)
0806 04:43:17.530113 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:17.530119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:17.530144 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:43:17.530267 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:17.530271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:17.530456 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:17.530643 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:17.531968 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.533287 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.534649 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.535416 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.536086 (+ 670us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:17.538839 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:17.538846 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:17.538859 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:17.538898 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:17.541857 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:17.542406 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:17.542413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:17.542414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:17.542566 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:17.542983 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:17.542987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:17.542989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:17.543502 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:17.545050 (+ 1548us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:17.545163 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 04:43:17.545327 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":568,"server-negotiator.queue_time_us":224,"spinlock_wait_cycles":3456,"thread_start_us":62,"threads_started":1}
I20260806 04:43:17.545634 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:17.529167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:42257 (local address 127.31.166.129:36053)
0806 04:43:17.529454 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:17.529469 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:17.529570 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:17.531848 (+ 2278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:17.531851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:17.532038 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:17.532136 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:17.532143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.534061 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.534064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:17.534975 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:17.534989 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.536856 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.536863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:17.536967 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:17.537912 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:43:17.537930 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:17.540818 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:17.543929 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:17.543936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:17.543941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:17.544224 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:17.544549 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:17.544552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:17.544554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:17.544638 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:17.545140 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:17.545143 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:17.545229 (+ 86us) client_negotiation.cc:769] Sending connection context
0806 04:43:17.545323 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 04:43:17.545456 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260806 04:43:17.545936 18080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:17.528689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:53657 (local address 127.31.166.129:42471)
0806 04:43:17.529617 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:17.529625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:17.529650 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:43:17.529797 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:17.529803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:17.530074 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:17.531155 (+ 1081us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:17.531611 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.532955 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.534433 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.534950 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.536309 (+ 1359us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:17.539738 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:17.539746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:17.539758 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:17.539786 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:17.541837 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:17.542425 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:17.542430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:17.542431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:17.542566 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:17.542978 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:17.542981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:17.542983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:17.543502 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:17.543649 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:17.545588 (+ 1939us) server_negotiation.cc:299] Negotiation successful
0806 04:43:17.545765 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":840,"thread_start_us":88,"threads_started":1}
I20260806 04:43:17.545665 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:17.528111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:42257 (local address 127.31.166.130:56387)
0806 04:43:17.528263 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:17.528301 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:17.528789 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:17.531344 (+ 2555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:17.531347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:17.531565 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:17.531720 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:17.531730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.534064 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.534067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:17.535169 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:17.535180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.536120 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.536125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:17.536253 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:17.536789 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:43:17.536812 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:17.539619 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:17.542660 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:17.542665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:17.542668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:17.542939 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:17.544280 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:17.544286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:17.544291 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:17.544399 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:17.545138 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:17.545141 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:17.545230 (+ 89us) client_negotiation.cc:769] Sending connection context
0806 04:43:17.545314 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 04:43:17.545456 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":73}
I20260806 04:43:17.546509 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:17.529179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:56387 (local address 127.31.166.131:42257)
0806 04:43:17.529510 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:17.529537 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:17.529581 (+ 44us) server_negotiation.cc:407] Connection header received
0806 04:43:17.530595 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:17.530604 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:17.531001 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:17.531957 (+ 956us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:17.532451 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.533918 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.535322 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.535950 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.536577 (+ 627us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:17.539834 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:17.539839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:17.539849 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:17.539876 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:17.542146 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:17.543988 (+ 1842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:17.543995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:17.544002 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:17.544154 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:17.544530 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:17.544535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:17.544536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:17.545015 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:17.545739 (+ 724us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:17.545915 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 04:43:17.546311 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":169,"server-negotiator.queue_time_us":211,"spinlock_wait_cycles":7680,"thread_start_us":78,"threads_started":1}
I20260806 04:43:17.547307 17738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9f5f1594914c9aa5369d0d17d3b972" candidate_uuid: "195bf04a5e674522a9648b8213e85fb1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1183be6c6cc74149b98167e49fb300d8" is_pre_election: true
I20260806 04:43:17.547400 17882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9f5f1594914c9aa5369d0d17d3b972" candidate_uuid: "1183be6c6cc74149b98167e49fb300d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "195bf04a5e674522a9648b8213e85fb1" is_pre_election: true
I20260806 04:43:17.547595 17882 raft_consensus.cc:2454] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1183be6c6cc74149b98167e49fb300d8 in term 1.
I20260806 04:43:17.547596 17738 raft_consensus.cc:2454] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195bf04a5e674522a9648b8213e85fb1 in term 1.
I20260806 04:43:17.547667 18083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:17.529240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:36053 (local address 127.31.166.131:42257)
0806 04:43:17.530304 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:17.530314 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:17.530331 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:43:17.531541 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:17.531547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:17.531601 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:17.532249 (+ 648us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:17.532420 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.533904 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.535174 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:17.535966 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:17.537588 (+ 1622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:17.540965 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:17.540970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:17.540974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:17.541031 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:17.543829 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:17.544359 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:17.544362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:17.544364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:17.544440 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:17.544750 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:17.544753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:17.544755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:17.545015 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:17.547311 (+ 2296us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:17.547390 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 04:43:17.547503 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":305,"server-negotiator.queue_time_us":984,"spinlock_wait_cycles":1920,"thread_start_us":162,"threads_started":1}
I20260806 04:43:17.548661 17663 leader_election.cc:302] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1183be6c6cc74149b98167e49fb300d8, 195bf04a5e674522a9648b8213e85fb1; no voters:
W20260806 04:43:17.548867 17809 leader_election.cc:341] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer edee860e107746ac9fb1601036495d9c (127.31.166.131:42257): Not found: Tablet not found: 1d9f5f1594914c9aa5369d0d17d3b972
I20260806 04:43:17.548964 18078 raft_consensus.cc:2781] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:43:17.549073 18078 raft_consensus.cc:482] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:43:17.549121 18078 raft_consensus.cc:3037] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:43:17.549117 17809 leader_election.cc:302] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1183be6c6cc74149b98167e49fb300d8, 195bf04a5e674522a9648b8213e85fb1; no voters: edee860e107746ac9fb1601036495d9c
W20260806 04:43:17.549250 17662 leader_election.cc:341] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer edee860e107746ac9fb1601036495d9c (127.31.166.131:42257): Not found: Tablet not found: 1d9f5f1594914c9aa5369d0d17d3b972
I20260806 04:43:17.549391 18077 raft_consensus.cc:2781] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 04:43:17.549468 18077 raft_consensus.cc:482] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:43:17.549513 18077 raft_consensus.cc:3037] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 04:43:17.552121 18077 raft_consensus.cc:504] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:43:17.552227 18078 raft_consensus.cc:504] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:43:17.552568 18077 leader_election.cc:288] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 2 election: Requested vote from peers edee860e107746ac9fb1601036495d9c (127.31.166.131:42257), 1183be6c6cc74149b98167e49fb300d8 (127.31.166.129:42471)
I20260806 04:43:17.552608 18078 leader_election.cc:288] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [CANDIDATE]: Term 2 election: Requested vote from peers 195bf04a5e674522a9648b8213e85fb1 (127.31.166.130:39161), edee860e107746ac9fb1601036495d9c (127.31.166.131:42257)
I20260806 04:43:17.553056 17738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9f5f1594914c9aa5369d0d17d3b972" candidate_uuid: "195bf04a5e674522a9648b8213e85fb1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1183be6c6cc74149b98167e49fb300d8"
I20260806 04:43:17.553056 17882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9f5f1594914c9aa5369d0d17d3b972" candidate_uuid: "1183be6c6cc74149b98167e49fb300d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "195bf04a5e674522a9648b8213e85fb1"
I20260806 04:43:17.553234 17882 raft_consensus.cc:2379] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1183be6c6cc74149b98167e49fb300d8 in current term 2: Already voted for candidate 195bf04a5e674522a9648b8213e85fb1 in this term.
I20260806 04:43:17.553234 17738 raft_consensus.cc:2379] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 195bf04a5e674522a9648b8213e85fb1 in current term 2: Already voted for candidate 1183be6c6cc74149b98167e49fb300d8 in this term.
W20260806 04:43:17.553303 17809 leader_election.cc:341] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer edee860e107746ac9fb1601036495d9c (127.31.166.131:42257): Not found: Tablet not found: 1d9f5f1594914c9aa5369d0d17d3b972
W20260806 04:43:17.553303 17662 leader_election.cc:341] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer edee860e107746ac9fb1601036495d9c (127.31.166.131:42257): Not found: Tablet not found: 1d9f5f1594914c9aa5369d0d17d3b972
I20260806 04:43:17.553644 17809 leader_election.cc:302] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 195bf04a5e674522a9648b8213e85fb1; no voters: 1183be6c6cc74149b98167e49fb300d8, edee860e107746ac9fb1601036495d9c
I20260806 04:43:17.553666 17663 leader_election.cc:302] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1183be6c6cc74149b98167e49fb300d8; no voters: 195bf04a5e674522a9648b8213e85fb1, edee860e107746ac9fb1601036495d9c
I20260806 04:43:17.553893 18077 raft_consensus.cc:2726] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:43:17.553893 18078 raft_consensus.cc:2726] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 04:43:17.946189 18081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:17.945886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:54507)
0806 04:43:17.946027 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:17.946100 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 04:43:18.101042 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:18.100582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:40481)
0806 04:43:18.100869 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:18.100926 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":105,"threads_started":1}
I20260806 04:43:18.287778 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:18.287303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:50791)
0806 04:43:18.287585 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:18.287660 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260806 04:43:18.947775 18088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:18.947163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:46275)
0806 04:43:18.947605 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:18.947672 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":88,"threads_started":1}
I20260806 04:43:19.102387 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:19.101832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:55021)
0806 04:43:19.102183 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:19.102244 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":81,"threads_started":1}
I20260806 04:43:19.289088 18090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:19.288636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:43017)
0806 04:43:19.288915 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:19.288991 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":79,"threads_started":1}
I20260806 04:43:19.460665 18091 raft_consensus.cc:482] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:43:19.460810 18091 raft_consensus.cc:504] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:43:19.461400 18091 leader_election.cc:288] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers edee860e107746ac9fb1601036495d9c (127.31.166.131:42257), 1183be6c6cc74149b98167e49fb300d8 (127.31.166.129:42471)
I20260806 04:43:19.461982 17738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9f5f1594914c9aa5369d0d17d3b972" candidate_uuid: "195bf04a5e674522a9648b8213e85fb1" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1183be6c6cc74149b98167e49fb300d8" is_pre_election: true
I20260806 04:43:19.462167 17738 raft_consensus.cc:2454] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 195bf04a5e674522a9648b8213e85fb1 in term 2.
W20260806 04:43:19.462327 17809 leader_election.cc:341] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 3 pre-election: Tablet error from VoteRequest() call to peer edee860e107746ac9fb1601036495d9c (127.31.166.131:42257): Not found: Tablet not found: 1d9f5f1594914c9aa5369d0d17d3b972
I20260806 04:43:19.462527 17809 leader_election.cc:302] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1183be6c6cc74149b98167e49fb300d8, 195bf04a5e674522a9648b8213e85fb1; no voters: edee860e107746ac9fb1601036495d9c
I20260806 04:43:19.462695 18091 raft_consensus.cc:2781] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 04:43:19.462760 18091 raft_consensus.cc:482] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:43:19.462802 18091 raft_consensus.cc:3037] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 04:43:19.463871 18091 raft_consensus.cc:504] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:43:19.464206 18091 leader_election.cc:288] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 3 election: Requested vote from peers edee860e107746ac9fb1601036495d9c (127.31.166.131:42257), 1183be6c6cc74149b98167e49fb300d8 (127.31.166.129:42471)
I20260806 04:43:19.464627 17738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d9f5f1594914c9aa5369d0d17d3b972" candidate_uuid: "195bf04a5e674522a9648b8213e85fb1" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "1183be6c6cc74149b98167e49fb300d8"
I20260806 04:43:19.464771 17738 raft_consensus.cc:3037] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 2 FOLLOWER]: Advancing to term 3
W20260806 04:43:19.464893 17809 leader_election.cc:341] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 3 election: Tablet error from VoteRequest() call to peer edee860e107746ac9fb1601036495d9c (127.31.166.131:42257): Not found: Tablet not found: 1d9f5f1594914c9aa5369d0d17d3b972
I20260806 04:43:19.465781 17738 raft_consensus.cc:2454] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 195bf04a5e674522a9648b8213e85fb1 in term 3.
I20260806 04:43:19.466101 17809 leader_election.cc:302] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1183be6c6cc74149b98167e49fb300d8, 195bf04a5e674522a9648b8213e85fb1; no voters: edee860e107746ac9fb1601036495d9c
I20260806 04:43:19.466290 18091 raft_consensus.cc:2781] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 04:43:19.466531 18091 raft_consensus.cc:686] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [term 3 LEADER]: Becoming Leader. State: Replica: 195bf04a5e674522a9648b8213e85fb1, State: Running, Role: LEADER
I20260806 04:43:19.466962 18091 consensus_queue.cc:237] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:43:19.467342 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:19.467026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:53031)
0806 04:43:19.467196 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:19.467262 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 04:43:19.878463 17738 raft_consensus.cc:1261] T 1d9f5f1594914c9aa5369d0d17d3b972 P 1183be6c6cc74149b98167e49fb300d8 [term 3 FOLLOWER]: Refusing update from remote peer 195bf04a5e674522a9648b8213e85fb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 04:43:19.879184 18091 consensus_queue.cc:1035] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 04:43:19.881532 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:19.881133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:55301)
0806 04:43:19.881367 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:19.881428 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
I20260806 04:43:19.884342 18096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:19.884043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:34019)
0806 04:43:19.884204 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:19.884266 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260806 04:43:19.884388 17809 consensus_peers.cc:487] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1 -> Peer edee860e107746ac9fb1601036495d9c (127.31.166.131:42257): Couldn't send request to peer edee860e107746ac9fb1601036495d9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d9f5f1594914c9aa5369d0d17d3b972. This is attempt 1: this message will repeat every 5th retry.
I20260806 04:43:19.884532 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:19.884175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:46829)
0806 04:43:19.884282 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:19.884380 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260806 04:43:20.290349 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:20.289846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:36881)
0806 04:43:20.290161 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:20.290228 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
I20260806 04:43:20.306757 18103 ts_tablet_manager.cc:927] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Initiating tablet copy from peer 195bf04a5e674522a9648b8213e85fb1 (127.31.166.130:39161)
I20260806 04:43:20.307461 18103 tablet_copy_client.cc:316] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: tablet copy: Beginning tablet copy session from remote peer at address 127.31.166.130:39161
Aug 06 04:43:20 dist-test-slave-vkjp krb5kdc[16396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991396, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.130@KRBTEST.COM
I20260806 04:43:20.316738 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:20.308024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:39161 (local address 127.31.166.131:38549)
0806 04:43:20.308172 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:20.308194 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:20.308322 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:20.308683 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:20.308688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:20.309092 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:20.309261 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:20.309270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:20.310234 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:20.310236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:20.310884 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:20.310893 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:20.311389 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:20.311392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:20.311473 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:20.311975 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:43:20.311987 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:20.313445 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:20.315336 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:20.315346 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:20.315352 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:20.315573 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:20.315816 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:20.315818 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:20.315819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:20.315900 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:20.316236 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:20.316241 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:20.316349 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 04:43:20.316478 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 04:43:20.316611 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 04:43:20.316783 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:20.308161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:38549 (local address 127.31.166.130:39161)
0806 04:43:20.308458 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:20.308465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:20.308479 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:43:20.308517 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:20.308519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:20.308558 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:20.308637 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:20.309408 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:20.310129 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:20.310993 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:20.311275 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:20.311359 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:20.313581 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:20.313584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:20.313586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:20.313610 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:20.315239 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:20.315657 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:20.315660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:20.315661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:20.315740 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:20.315974 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:20.315976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:20.315977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:20.316149 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:20.316223 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:20.316534 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 04:43:20.316634 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":121,"threads_started":1}
I20260806 04:43:20.317363 17903 tablet_copy_service.cc:140] P 195bf04a5e674522a9648b8213e85fb1: Received BeginTabletCopySession request for tablet 1d9f5f1594914c9aa5369d0d17d3b972 from peer edee860e107746ac9fb1601036495d9c ({username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:38549)
I20260806 04:43:20.317526 17903 tablet_copy_service.cc:161] P 195bf04a5e674522a9648b8213e85fb1: Beginning new tablet copy session on tablet 1d9f5f1594914c9aa5369d0d17d3b972 from peer edee860e107746ac9fb1601036495d9c at {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:38549: session id = edee860e107746ac9fb1601036495d9c-1d9f5f1594914c9aa5369d0d17d3b972
I20260806 04:43:20.318775 17903 tablet_copy_source_session.cc:205] T 1d9f5f1594914c9aa5369d0d17d3b972 P 195bf04a5e674522a9648b8213e85fb1: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 04:43:20.322620 18103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d9f5f1594914c9aa5369d0d17d3b972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:20.329080 18103 tablet_copy_client.cc:791] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: tablet copy: Starting download of 0 data blocks...
I20260806 04:43:20.329293 18103 tablet_copy_client.cc:655] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: tablet copy: Starting download of 1 WAL segments...
I20260806 04:43:20.331192 18103 tablet_copy_client.cc:531] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 04:43:20.333122 18103 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Bootstrap starting.
I20260806 04:43:20.338578 18103 log.cc:824] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:20.345167 18103 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 04:43:20.345494 18103 tablet_bootstrap.cc:492] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Bootstrap complete.
I20260806 04:43:20.345816 18103 ts_tablet_manager.cc:1397] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260806 04:43:20.348142 18103 raft_consensus.cc:348] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:43:20.348400 18103 raft_consensus.cc:729] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: edee860e107746ac9fb1601036495d9c, State: Initialized, Role: FOLLOWER
I20260806 04:43:20.348830 18103 consensus_queue.cc:260] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "195bf04a5e674522a9648b8213e85fb1" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39161 } } peers { permanent_uuid: "edee860e107746ac9fb1601036495d9c" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 42257 } } peers { permanent_uuid: "1183be6c6cc74149b98167e49fb300d8" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 42471 } }
I20260806 04:43:20.349797 18103 ts_tablet_manager.cc:1428] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20260806 04:43:20.350142 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:20.349850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:43353)
0806 04:43:20.349975 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:20.350038 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 04:43:20.350581 17903 tablet_copy_service.cc:342] P 195bf04a5e674522a9648b8213e85fb1: Request end of tablet copy session edee860e107746ac9fb1601036495d9c-1d9f5f1594914c9aa5369d0d17d3b972 received from {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:38549
I20260806 04:43:20.350704 17903 tablet_copy_service.cc:433] P 195bf04a5e674522a9648b8213e85fb1: ending tablet copy session edee860e107746ac9fb1601036495d9c-1d9f5f1594914c9aa5369d0d17d3b972 on tablet 1d9f5f1594914c9aa5369d0d17d3b972 with peer edee860e107746ac9fb1601036495d9c
I20260806 04:43:20.350939 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:20.350590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:33503)
0806 04:43:20.350821 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:20.350878 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260806 04:43:20.885535 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:20.885156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:43979)
0806 04:43:20.885396 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:20.885448 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":90,"threads_started":1}
I20260806 04:43:20.885677 18109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:20.885214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:44955)
0806 04:43:20.885499 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:20.885557 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":105,"threads_started":1}
I20260806 04:43:20.917662 18109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:20.917327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:43543)
0806 04:43:20.917494 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:20.917562 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 04:43:20.917829 18022 raft_consensus.cc:1203] T 1d9f5f1594914c9aa5369d0d17d3b972 P edee860e107746ac9fb1601036495d9c [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[3.3-3.3] Dedup: 3.3->[]
I20260806 04:43:20.919137 18113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:20.918705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:45533)
0806 04:43:20.918924 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:20.919001 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":77,"threads_started":1}
I20260806 04:43:21.886816 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:21.886399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:41361)
0806 04:43:21.886659 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:21.886718 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260806 04:43:21.919242 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:21.918667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:47197)
0806 04:43:21.918967 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:21.919052 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":75,"threads_started":1}
I20260806 04:43:21.920138 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:21.919857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:50299)
0806 04:43:21.920006 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:21.920068 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 04:43:21.920542 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:21.920136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:52657)
0806 04:43:21.920416 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:21.920466 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":73,"threads_started":1}
I20260806 04:43:22.888098 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:22.887654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:43887)
0806 04:43:22.887939 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:22.888003 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":125,"threads_started":1}
I20260806 04:43:22.921844 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:22.921338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:55227)
0806 04:43:22.921621 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:22.921714 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":103,"threads_started":1}
I20260806 04:43:22.921927 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:22.921111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:45919)
0806 04:43:22.921723 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:22.921809 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":93,"threads_started":1}
I20260806 04:43:23.889235 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:23.888835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:42973)
0806 04:43:23.889084 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:23.889134 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":88,"threads_started":1}
I20260806 04:43:23.923209 18121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:23.922681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:45289)
0806 04:43:23.923015 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:23.923096 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":82,"threads_started":1}
I20260806 04:43:23.923209 18122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:23.922662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:59111)
0806 04:43:23.923015 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:23.923099 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":96,"threads_started":1}
I20260806 04:43:24.890615 18123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:24.890064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:55637)
0806 04:43:24.890359 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:24.890453 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260806 04:43:24.924486 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:24.924004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:51953)
0806 04:43:24.924298 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:24.924386 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":108,"threads_started":1}
I20260806 04:43:24.924515 18125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:24.924003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:47927)
0806 04:43:24.924362 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:24.924428 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":106,"threads_started":1}
I20260806 04:43:25.892028 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:25.891542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:54009)
0806 04:43:25.891871 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:25.891933 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":123,"threads_started":1}
I20260806 04:43:25.925696 18127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:25.925256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:33727)
0806 04:43:25.925535 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:25.925601 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":83,"threads_started":1}
I20260806 04:43:25.925734 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:25.925311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:33951)
0806 04:43:25.925592 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:25.925646 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":105,"threads_started":1}
I20260806 04:43:26.893389 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:26.892912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:52839)
0806 04:43:26.893206 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:26.893273 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":80,"threads_started":1}
I20260806 04:43:26.927199 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:26.926689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:56421)
0806 04:43:26.926991 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:26.927053 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":125,"threads_started":1}
I20260806 04:43:26.927214 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:26.926646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:59837)
0806 04:43:26.926963 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:26.927041 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":81,"threads_started":1}
I20260806 04:43:27.894773 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:27.894297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:35939)
0806 04:43:27.894585 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:27.894672 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":81,"threads_started":1}
I20260806 04:43:27.928563 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:27.928056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:48671)
0806 04:43:27.928400 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:27.928464 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":92,"threads_started":1}
I20260806 04:43:27.928586 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:27.928104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:52685)
0806 04:43:27.928442 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:27.928499 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
I20260806 04:43:28.896252 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:28.895639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:53145)
0806 04:43:28.895997 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:28.896073 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":92,"threads_started":1}
I20260806 04:43:28.929997 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:28.929395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:58929)
0806 04:43:28.929773 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:28.929852 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":89,"threads_started":1}
I20260806 04:43:28.930526 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:28.929386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:59695)
0806 04:43:28.930338 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:28.930408 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":838,"thread_start_us":104,"threads_started":1}
I20260806 04:43:29.897686 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:29.897235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:41643)
0806 04:43:29.897509 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:29.897574 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
I20260806 04:43:29.931413 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:29.930794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:42223)
0806 04:43:29.931169 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:29.931269 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":147,"threads_started":1}
I20260806 04:43:29.931878 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:29.931452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:53353)
0806 04:43:29.931735 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:29.931795 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":75,"threads_started":1}
I20260806 04:43:30.899240 18142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:30.898699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.129:41107)
0806 04:43:30.899012 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:30.899127 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":103,"threads_started":1}
I20260806 04:43:30.932868 18143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:30.932377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.131:52111)
0806 04:43:30.932693 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:30.932761 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":128,"threads_started":1}
I20260806 04:43:30.933095 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:30.932662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33009 (local address 127.31.166.130:50865)
0806 04:43:30.932908 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:30.932987 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.190:33009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":57,"threads_started":1}
I20260806 04:43:31.288149 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 17602
I20260806 04:43:31.295230 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 17788
I20260806 04:43:31.302901 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 17934
2026-08-06T04:43:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59428 ms)
[----------] 30 tests from SecurityITest (251158 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:31 dist-test-slave-vkjp krb5kdc[18147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:31 dist-test-slave-vkjp krb5kdc[18147](info): set up 2 sockets
Aug 06 04:43:31 dist-test-slave-vkjp krb5kdc[18147](info): commencing operation
krb5kdc: starting...
W20260806 04:43:32.442065 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:43:32 dist-test-slave-vkjp krb5kdc[18147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:43:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:43:32.571760 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:43183
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:43183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:32.682452 18161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:32.682749 18161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:32.682823 18161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:32.686868 18161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:32.686964 18161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:32.686992 18161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:32.687022 18161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:32.687075 18161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:32.692401 18161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:43183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:43183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:32.693506 18161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:32.694556 18161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:32.703269 18168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:32.703280 18167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:32.703305 18161 server_base.cc:1034] running on GCE node
W20260806 04:43:32.703254 18170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:32.704156 18161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:32.705288 18161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:32.706454 18161 hybrid_clock.cc:648] HybridClock initialized: now 1785991412706437 us; error 31 us; skew 500 ppm
Aug 06 04:43:32 dist-test-slave-vkjp krb5kdc[18147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991412, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:32.709285 18161 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:43:32.710196 18161 webserver.cc:460] Webserver started at http://127.31.166.190:32965/ using document root <none> and password file <none>
I20260806 04:43:32.710750 18161 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:32.710842 18161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:32.711119 18161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:32.713184 18161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "40db9c16c6a6421888bdfd8a0d317375"
format_stamp: "Formatted at 2026-08-06 04:43:32 on dist-test-slave-vkjp"
server_key: "3b80b0c3c5e810cdd7e409b4b8568ca6"
server_key_iv: "8b40386d89ca22eca78894db8754fc0c"
server_key_version: "encryptionkey@0"
I20260806 04:43:32.713750 18161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "40db9c16c6a6421888bdfd8a0d317375"
format_stamp: "Formatted at 2026-08-06 04:43:32 on dist-test-slave-vkjp"
server_key: "3b80b0c3c5e810cdd7e409b4b8568ca6"
server_key_iv: "8b40386d89ca22eca78894db8754fc0c"
server_key_version: "encryptionkey@0"
I20260806 04:43:32.717499 18161 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:43:32.720088 18178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:32.721297 18161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:43:32.721438 18161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "40db9c16c6a6421888bdfd8a0d317375"
format_stamp: "Formatted at 2026-08-06 04:43:32 on dist-test-slave-vkjp"
server_key: "3b80b0c3c5e810cdd7e409b4b8568ca6"
server_key_iv: "8b40386d89ca22eca78894db8754fc0c"
server_key_version: "encryptionkey@0"
I20260806 04:43:32.721557 18161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:32.741425 18161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:32.744377 18161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:32.744586 18161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:32.752032 18161 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:43183
I20260806 04:43:32.752058 18234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:43183 every 8 connection(s)
I20260806 04:43:32.753021 18161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:32.755764 18235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:32.758924 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 18161
I20260806 04:43:32.759063 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:32.759402 32410 external_mini_cluster.cc:1428] Setting key 11aa9ae9efc23ae7fdce239e927ca68c
I20260806 04:43:32.761750 18235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375: Bootstrap starting.
I20260806 04:43:32.763968 18235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:32.764775 18235 log.cc:824] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:32.766637 18235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375: No bootstrap required, opened a new log
Aug 06 04:43:32 dist-test-slave-vkjp krb5kdc[18147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:43:32.770434 18235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40db9c16c6a6421888bdfd8a0d317375" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43183 } }
I20260806 04:43:32.770645 18235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:32.770715 18235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40db9c16c6a6421888bdfd8a0d317375, State: Initialized, Role: FOLLOWER
I20260806 04:43:32.771224 18235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40db9c16c6a6421888bdfd8a0d317375" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43183 } }
I20260806 04:43:32.771360 18235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:32.771445 18235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:32.771546 18235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:32.772404 18235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40db9c16c6a6421888bdfd8a0d317375" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43183 } }
I20260806 04:43:32.772740 18235 leader_election.cc:302] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40db9c16c6a6421888bdfd8a0d317375; no voters:
I20260806 04:43:32.773084 18235 leader_election.cc:288] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:32.773156 18240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:32.773510 18240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [term 1 LEADER]: Becoming Leader. State: Replica: 40db9c16c6a6421888bdfd8a0d317375, State: Running, Role: LEADER
I20260806 04:43:32.773962 18240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40db9c16c6a6421888bdfd8a0d317375" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43183 } }
I20260806 04:43:32.774334 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:32.760843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56078 (local address 127.31.166.190:43183)
0806 04:43:32.761291 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:32.761307 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:32.761341 (+ 34us) server_negotiation.cc:407] Connection header received
0806 04:43:32.762033 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:32.762052 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:32.762360 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:32.762679 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:32.763571 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:32.764714 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:32.765823 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:32.766283 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:32.766476 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:32.769206 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:32.769211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:32.769223 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:32.769250 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:32.771135 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:32.771686 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:32.771691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:32.771695 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:32.771802 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:32.772096 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:32.772100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:32.772102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:32.772480 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:32.772684 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:32.773024 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 04:43:32.773358 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":153,"threads_started":1}
I20260806 04:43:32.774819 18235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:32.775949 18240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40db9c16c6a6421888bdfd8a0d317375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40db9c16c6a6421888bdfd8a0d317375" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43183 } } }
I20260806 04:43:32.776068 18240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:32.777184 18241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40db9c16c6a6421888bdfd8a0d317375. Latest consensus state: current_term: 1 leader_uuid: "40db9c16c6a6421888bdfd8a0d317375" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40db9c16c6a6421888bdfd8a0d317375" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43183 } } }
I20260806 04:43:32.777308 18241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375 [sys.catalog]: This master's current role is: LEADER
W20260806 04:43:32.780006 18257 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:43:32.780071 18257 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:43:32.780138 18249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:32.781102 18249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:32.786710 18249 catalog_manager.cc:1347] Generated new cluster ID: da5e71906fe448ad979180442e528a2a
I20260806 04:43:32.786827 18249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:32.807655 18249 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:32.808960 18249 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:32.820168 18249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375: Generated new TSK 0
I20260806 04:43:32.821399 18249 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:43:32.856153 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43183
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:32.974257 18263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:32.974596 18263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:32.974649 18263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:32.979422 18263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:32.979494 18263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:32.979557 18263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:43:32.985097 18263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:32.986109 18263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:32.987133 18263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:32.997025 18268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:32.997032 18269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:32.997038 18271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:32.997918 18263 server_base.cc:1034] running on GCE node
I20260806 04:43:32.998381 18263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:32.999133 18263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:33.000330 18263 hybrid_clock.cc:648] HybridClock initialized: now 1785991413000320 us; error 29 us; skew 500 ppm
Aug 06 04:43:32 dist-test-slave-vkjp krb5kdc[18147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991412, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:33.003612 18263 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:43:33.004457 18263 webserver.cc:460] Webserver started at http://127.31.166.129:41553/ using document root <none> and password file <none>
I20260806 04:43:33.004984 18263 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:33.005043 18263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:33.005210 18263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:33.007320 18263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "58fc2b24877144eb9476288668215605"
format_stamp: "Formatted at 2026-08-06 04:43:32 on dist-test-slave-vkjp"
server_key: "ed5ac7213f14565f585979c96193f539"
server_key_iv: "6c5345f86b7c6a969d2dfb37df430a8d"
server_key_version: "encryptionkey@0"
I20260806 04:43:33.007856 18263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "58fc2b24877144eb9476288668215605"
format_stamp: "Formatted at 2026-08-06 04:43:32 on dist-test-slave-vkjp"
server_key: "ed5ac7213f14565f585979c96193f539"
server_key_iv: "6c5345f86b7c6a969d2dfb37df430a8d"
server_key_version: "encryptionkey@0"
I20260806 04:43:33.011612 18263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:43:33.014038 18278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:33.015483 18263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:43:33.015631 18263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "58fc2b24877144eb9476288668215605"
format_stamp: "Formatted at 2026-08-06 04:43:32 on dist-test-slave-vkjp"
server_key: "ed5ac7213f14565f585979c96193f539"
server_key_iv: "6c5345f86b7c6a969d2dfb37df430a8d"
server_key_version: "encryptionkey@0"
I20260806 04:43:33.015754 18263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:33.037091 18263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:33.040573 18263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:33.040808 18263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:33.041479 18263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:33.042397 18263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:33.042477 18263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:33.042557 18263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:33.042613 18263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:33.052564 18263 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:39793
I20260806 04:43:33.052604 18397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:39793 every 8 connection(s)
I20260806 04:43:33.053866 18263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:43:33.063715 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 18263
I20260806 04:43:33.063851 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:43:33.064268 32410 external_mini_cluster.cc:1428] Setting key c770ed0b153e7c75727353e34bb9df13
Aug 06 04:43:33 dist-test-slave-vkjp krb5kdc[18147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991412, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:43:33.070772 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:33.056327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:41277 (local address 127.31.166.190:43183)
0806 04:43:33.056508 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:33.056513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:33.057548 (+ 1035us) server_negotiation.cc:407] Connection header received
0806 04:43:33.059742 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:33.059747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:33.059815 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:33.059910 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:33.061320 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.062042 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.063639 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.063988 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.064088 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:33.067019 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:33.067023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:33.067025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:33.067052 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:33.068645 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:33.069186 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:33.069190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:33.069192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:33.069278 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:33.069631 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:33.069636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:33.069638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:33.069834 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:33.069949 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:33.070466 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 04:43:33.070598 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 04:43:33.071660 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:33.056678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43183 (local address 127.31.166.129:41277)
0806 04:43:33.057360 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:33.057427 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:33.059455 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:33.060103 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:33.060109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:33.060478 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:33.061133 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:33.061155 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.062179 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.062185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:33.063485 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:33.063498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.064164 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.064170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:33.064396 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:33.065145 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:43:33.065170 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:33.066846 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:33.068770 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:33.068776 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:33.068791 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:33.069076 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:33.069384 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:33.069387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:33.069389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:33.069533 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:33.069951 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:33.069958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:33.070214 (+ 256us) client_negotiation.cc:769] Sending connection context
0806 04:43:33.070426 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 04:43:33.070751 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":129,"threads_started":1}
I20260806 04:43:33.072923 18398 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43183
I20260806 04:43:33.073205 18398 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:33.073917 18398 heartbeater.cc:511] Master 127.31.166.190:43183 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
I20260806 04:43:33.075963 18195 ts_manager.cc:194] Registered new tserver with Master: 58fc2b24877144eb9476288668215605 (127.31.166.129:39793)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
I20260806 04:43:33.077410 18195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:41277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:43:33.096107 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43183
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:33.210179 18404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:33.210703 18404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:33.210835 18404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:33.217499 18404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:33.217566 18404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:33.217665 18404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:43:33.223505 18404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:33.224678 18404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:33.225847 18404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:33.237213 18410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:33.237301 18409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:33.237527 18412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:33.237895 18404 server_base.cc:1034] running on GCE node
I20260806 04:43:33.238312 18404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:33.239020 18404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:33.240235 18404 hybrid_clock.cc:648] HybridClock initialized: now 1785991413240207 us; error 42 us; skew 500 ppm
Aug 06 04:43:33 dist-test-slave-vkjp krb5kdc[18147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991413, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:33.243432 18404 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:43:33.244632 18404 webserver.cc:460] Webserver started at http://127.31.166.130:40991/ using document root <none> and password file <none>
I20260806 04:43:33.245492 18404 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:33.245594 18404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:33.245858 18404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:33.249464 18404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "be21577bc87d47c1a385c341db03ac2a"
format_stamp: "Formatted at 2026-08-06 04:43:33 on dist-test-slave-vkjp"
server_key: "231822aa2191d0afb6123844157398b7"
server_key_iv: "25eacbe540e3cb64745cf70335705995"
server_key_version: "encryptionkey@0"
I20260806 04:43:33.250216 18404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "be21577bc87d47c1a385c341db03ac2a"
format_stamp: "Formatted at 2026-08-06 04:43:33 on dist-test-slave-vkjp"
server_key: "231822aa2191d0afb6123844157398b7"
server_key_iv: "25eacbe540e3cb64745cf70335705995"
server_key_version: "encryptionkey@0"
I20260806 04:43:33.255663 18404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260806 04:43:33.259196 18419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:33.260828 18404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:43:33.260964 18404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "be21577bc87d47c1a385c341db03ac2a"
format_stamp: "Formatted at 2026-08-06 04:43:33 on dist-test-slave-vkjp"
server_key: "231822aa2191d0afb6123844157398b7"
server_key_iv: "25eacbe540e3cb64745cf70335705995"
server_key_version: "encryptionkey@0"
I20260806 04:43:33.261118 18404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:33.284534 18404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:33.289002 18404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:33.289264 18404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:33.290160 18404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:33.291442 18404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:33.291518 18404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:33.291589 18404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:33.291630 18404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:33.305015 18404 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:42419
I20260806 04:43:33.305068 18534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:42419 every 8 connection(s)
I20260806 04:43:33.306416 18404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:43:33.313694 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 18404
I20260806 04:43:33.313879 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:43:33.314189 32410 external_mini_cluster.cc:1428] Setting key 093208800bbbfa859c38126e3f59b29d
Aug 06 04:43:33 dist-test-slave-vkjp krb5kdc[18147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991413, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:43:33.323040 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:33.308435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:41345 (local address 127.31.166.190:43183)
0806 04:43:33.308612 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:33.308618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:33.309334 (+ 716us) server_negotiation.cc:407] Connection header received
0806 04:43:33.310689 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:33.310695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:33.310764 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:33.310940 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:33.312127 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.313143 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.314861 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.315360 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.315495 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:33.318222 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:33.318229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:33.318234 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:33.318272 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:33.320745 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:33.321437 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:33.321440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:33.321442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:33.321527 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:33.321881 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:33.321884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:33.321886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:33.322076 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:33.322206 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:33.322725 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 04:43:33.322854 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:43:33.323958 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:33.308695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43183 (local address 127.31.166.130:41345)
0806 04:43:33.309162 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:33.309216 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:33.310476 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:33.311129 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:33.311138 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:33.311507 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:33.311972 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:33.311985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.313276 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.313278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:33.314683 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:33.314693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.315642 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.315647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:33.315807 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:33.316460 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:43:33.316477 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:33.318013 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:33.320905 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:33.320913 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:33.320930 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:33.321313 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:33.321624 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:33.321628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:33.321630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:33.321774 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:33.322215 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:33.322223 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:33.322478 (+ 255us) client_negotiation.cc:769] Sending connection context
0806 04:43:33.322668 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 04:43:33.322990 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":150,"threads_started":1}
I20260806 04:43:33.325136 18535 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43183
I20260806 04:43:33.325389 18535 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:33.325964 18535 heartbeater.cc:511] Master 127.31.166.190:43183 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:43:33.326998 18195 ts_manager.cc:194] Registered new tserver with Master: be21577bc87d47c1a385c341db03ac2a (127.31.166.130:42419)
I20260806 04:43:33.327728 18195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:41345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:43:33.349797 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43183
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:46203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:33.469058 18542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:33.469347 18542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:33.469393 18542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:33.473685 18542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:33.473757 18542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:33.473858 18542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:43:33.479630 18542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:46203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:43183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:33.480731 18542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:33.481809 18542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:33.491649 18549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:33.491779 18542 server_base.cc:1034] running on GCE node
W20260806 04:43:33.491664 18548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:33.491703 18551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:33.492411 18542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:33.493062 18542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:33.494262 18542 hybrid_clock.cc:648] HybridClock initialized: now 1785991413494229 us; error 53 us; skew 500 ppm
Aug 06 04:43:33 dist-test-slave-vkjp krb5kdc[18147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991413, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:33.497232 18542 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:43:33.498147 18542 webserver.cc:460] Webserver started at http://127.31.166.131:39889/ using document root <none> and password file <none>
I20260806 04:43:33.498726 18542 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:33.498821 18542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:33.499065 18542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:33.501175 18542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "77cc566d595d4781b977393a65d44d3e"
format_stamp: "Formatted at 2026-08-06 04:43:33 on dist-test-slave-vkjp"
server_key: "12e09b26cf7c2746bdf81ce2f02f4ae7"
server_key_iv: "0cd8fa7e4bb0aab1ef70182ee65683b1"
server_key_version: "encryptionkey@0"
I20260806 04:43:33.501755 18542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "77cc566d595d4781b977393a65d44d3e"
format_stamp: "Formatted at 2026-08-06 04:43:33 on dist-test-slave-vkjp"
server_key: "12e09b26cf7c2746bdf81ce2f02f4ae7"
server_key_iv: "0cd8fa7e4bb0aab1ef70182ee65683b1"
server_key_version: "encryptionkey@0"
I20260806 04:43:33.505586 18542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260806 04:43:33.508114 18558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:33.509333 18542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:43:33.509472 18542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "77cc566d595d4781b977393a65d44d3e"
format_stamp: "Formatted at 2026-08-06 04:43:33 on dist-test-slave-vkjp"
server_key: "12e09b26cf7c2746bdf81ce2f02f4ae7"
server_key_iv: "0cd8fa7e4bb0aab1ef70182ee65683b1"
server_key_version: "encryptionkey@0"
I20260806 04:43:33.509588 18542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:33.540668 18542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:33.544176 18542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:33.544409 18542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:33.545042 18542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:33.545898 18542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:33.545974 18542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:33.546046 18542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:33.546095 18542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:33.555681 18542 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:36253
I20260806 04:43:33.555706 18678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:36253 every 8 connection(s)
I20260806 04:43:33.556604 18542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:43:33.557947 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 18542
I20260806 04:43:33.558095 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:43:33.558363 32410 external_mini_cluster.cc:1428] Setting key 38cab10ce5560d6c97d236c8da0560cd
Aug 06 04:43:33 dist-test-slave-vkjp krb5kdc[18147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991413, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:43:33.570520 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:33.558731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:48947 (local address 127.31.166.190:43183)
0806 04:43:33.558938 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:33.558942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:33.559912 (+ 970us) server_negotiation.cc:407] Connection header received
0806 04:43:33.561277 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:33.561280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:33.561326 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:33.561404 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:33.562572 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.563137 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.564255 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.564484 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.564566 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:33.567143 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:33.567147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:33.567149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:33.567175 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:33.568518 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:33.569011 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:33.569014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:33.569015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:33.569100 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:33.569437 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:33.569439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:33.569441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:33.569651 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:33.569741 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:33.570235 (+ 494us) server_negotiation.cc:299] Negotiation successful
0806 04:43:33.570391 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 04:43:33.571532 18682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:33.559244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:43183 (local address 127.31.166.131:48947)
0806 04:43:33.559751 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:33.559813 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:33.561092 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:33.561565 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:33.561572 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:33.561946 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:33.562443 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:33.562457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.563260 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.563263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:33.564144 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:33.564153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.564640 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.564644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:33.564817 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:33.565369 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:43:33.565385 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:33.566887 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:33.568624 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:33.568629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:33.568641 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:33.568924 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:33.569181 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:33.569183 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:33.569184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:33.569319 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:33.569750 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:33.569754 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:33.570014 (+ 260us) client_negotiation.cc:769] Sending connection context
0806 04:43:33.570173 (+ 159us) client_negotiation.cc:240] Negotiation successful
0806 04:43:33.570437 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":149,"threads_started":1}
I20260806 04:43:33.572732 18679 heartbeater.cc:347] Connected to a master server at 127.31.166.190:43183
I20260806 04:43:33.572947 18679 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:33.573393 18679 heartbeater.cc:511] Master 127.31.166.190:43183 requested a full tablet report, sending...
I20260806 04:43:33.574359 18194 ts_manager.cc:194] Registered new tserver with Master: 77cc566d595d4781b977393a65d44d3e (127.31.166.131:36253)
I20260806 04:43:33.574997 18194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:48947
I20260806 04:43:33.584328 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:43:33.595621 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:33.586355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.31.166.190:43183)
0806 04:43:33.586541 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:33.586547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:33.586755 (+ 208us) server_negotiation.cc:407] Connection header received
0806 04:43:33.586948 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:33.586951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:33.587005 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:33.587113 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:33.587844 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.588379 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.589241 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.589490 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.589641 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:33.591143 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:33.591147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:33.591150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:33.591180 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:33.593475 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:33.594023 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:33.594028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:33.594032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:33.594165 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:33.594547 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:33.594550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:33.594552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:33.594832 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:33.594975 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:33.595381 (+ 406us) server_negotiation.cc:299] Negotiation successful
0806 04:43:33.595497 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 04:43:33.598862 18194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:43:33.601974 18194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:43:33.617041 18693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:33.611018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:39793 (local address 127.0.0.1:37170)
0806 04:43:33.612120 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:33.612130 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:33.612207 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:33.612623 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:33.612626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:33.612646 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:33.612739 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:33.612748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.614077 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.614080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:33.615453 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:33.615463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.616476 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.616481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:33.616588 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:33.616625 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 04:43:33.616740 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 04:43:33.616869 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"mutex_wait_us":51,"thread_start_us":616,"threads_started":1}
I20260806 04:43:33.617071 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:33.611370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.31.166.129:39793)
0806 04:43:33.611891 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:33.611899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:33.612221 (+ 322us) server_negotiation.cc:407] Connection header received
0806 04:43:33.612328 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:33.612333 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:33.612510 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:33.612638 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:33.612963 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.613900 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.615577 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.616325 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.616464 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:33.616600 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:33.616787 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 04:43:33.616894 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":87,"threads_started":1}
I20260806 04:43:33.617457 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:33.611490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.31.166.131:36253)
0806 04:43:33.611995 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:33.612003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:33.612021 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:43:33.612083 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:33.612087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:33.612294 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:33.612446 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:33.612765 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.613970 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.615787 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.616459 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.616576 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:33.616702 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:33.617193 (+ 491us) server_negotiation.cc:299] Negotiation successful
0806 04:43:33.617298 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":136,"threads_started":1}
I20260806 04:43:33.617969 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:33.610784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.31.166.130:42419)
0806 04:43:33.612442 (+ 1658us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:33.612454 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:33.612475 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:43:33.612545 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:33.612550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:33.612791 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:33.612951 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:33.613166 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.614282 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.615975 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.616847 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.617530 (+ 683us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:33.617662 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:33.617716 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 04:43:33.617809 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1535,"thread_start_us":149,"threads_started":1}
I20260806 04:43:33.617100 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:33.610741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:36253 (local address 127.0.0.1:59912)
0806 04:43:33.611753 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:33.611766 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:33.611847 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:33.612460 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:33.612466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:33.612491 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:33.612626 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:33.612634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.614284 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.614288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:33.615650 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:33.615660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.616764 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.616767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:33.616846 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:33.616866 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 04:43:33.616932 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:43:33.616987 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":4,"thread_start_us":142,"threads_started":1}
I20260806 04:43:33.617391 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:33.610652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:42419 (local address 127.0.0.1:43298)
0806 04:43:33.611306 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:33.611369 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:33.611506 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:33.612938 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:33.612943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:33.612962 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:33.613051 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:33.613056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.614689 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.614692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:33.615829 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:33.615838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:33.616998 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:33.617001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:33.617104 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:33.617130 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 04:43:33.617186 (+ 56us) client_negotiation.cc:240] Negotiation successful
0806 04:43:33.617259 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260806 04:43:33.620829 18467 tablet_service.cc:1467] Processing CreateTablet for tablet 1789bbc44f904ca2ba1cb78e2ef76f56 (DEFAULT_TABLE table=test-table [id=5b7ac29d95c14036aef0a00c869f4b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:33.620812 18609 tablet_service.cc:1467] Processing CreateTablet for tablet 1789bbc44f904ca2ba1cb78e2ef76f56 (DEFAULT_TABLE table=test-table [id=5b7ac29d95c14036aef0a00c869f4b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:33.620765 18329 tablet_service.cc:1467] Processing CreateTablet for tablet 1789bbc44f904ca2ba1cb78e2ef76f56 (DEFAULT_TABLE table=test-table [id=5b7ac29d95c14036aef0a00c869f4b98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:33.621819 18467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1789bbc44f904ca2ba1cb78e2ef76f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:33.621832 18329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1789bbc44f904ca2ba1cb78e2ef76f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:33.621832 18609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1789bbc44f904ca2ba1cb78e2ef76f56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:33.629098 18698 tablet_bootstrap.cc:492] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a: Bootstrap starting.
I20260806 04:43:33.629756 18700 tablet_bootstrap.cc:492] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605: Bootstrap starting.
I20260806 04:43:33.629756 18699 tablet_bootstrap.cc:492] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e: Bootstrap starting.
I20260806 04:43:33.631290 18698 tablet_bootstrap.cc:654] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:33.631948 18700 tablet_bootstrap.cc:654] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:33.631948 18699 tablet_bootstrap.cc:654] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:33.632625 18698 log.cc:824] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:33.632850 18700 log.cc:824] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:33.633234 18699 log.cc:824] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:33.634822 18698 tablet_bootstrap.cc:492] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a: No bootstrap required, opened a new log
I20260806 04:43:33.635048 18698 ts_tablet_manager.cc:1397] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 04:43:33.635263 18699 tablet_bootstrap.cc:492] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e: No bootstrap required, opened a new log
I20260806 04:43:33.635457 18699 ts_tablet_manager.cc:1397] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 04:43:33.635696 18700 tablet_bootstrap.cc:492] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605: No bootstrap required, opened a new log
I20260806 04:43:33.635880 18700 ts_tablet_manager.cc:1397] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:43:33.638233 18698 raft_consensus.cc:348] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be21577bc87d47c1a385c341db03ac2a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42419 } } peers { permanent_uuid: "77cc566d595d4781b977393a65d44d3e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 36253 } } peers { permanent_uuid: "58fc2b24877144eb9476288668215605" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39793 } }
I20260806 04:43:33.638554 18698 raft_consensus.cc:374] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:33.638612 18698 raft_consensus.cc:729] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be21577bc87d47c1a385c341db03ac2a, State: Initialized, Role: FOLLOWER
I20260806 04:43:33.639165 18698 consensus_queue.cc:260] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be21577bc87d47c1a385c341db03ac2a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42419 } } peers { permanent_uuid: "77cc566d595d4781b977393a65d44d3e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 36253 } } peers { permanent_uuid: "58fc2b24877144eb9476288668215605" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39793 } }
I20260806 04:43:33.639515 18699 raft_consensus.cc:348] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be21577bc87d47c1a385c341db03ac2a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42419 } } peers { permanent_uuid: "77cc566d595d4781b977393a65d44d3e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 36253 } } peers { permanent_uuid: "58fc2b24877144eb9476288668215605" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39793 } }
I20260806 04:43:33.639995 18535 heartbeater.cc:503] Master 127.31.166.190:43183 was elected leader, sending a full tablet report...
I20260806 04:43:33.640269 18698 ts_tablet_manager.cc:1428] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:43:33.640318 18700 raft_consensus.cc:348] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be21577bc87d47c1a385c341db03ac2a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42419 } } peers { permanent_uuid: "77cc566d595d4781b977393a65d44d3e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 36253 } } peers { permanent_uuid: "58fc2b24877144eb9476288668215605" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39793 } }
I20260806 04:43:33.640573 18700 raft_consensus.cc:374] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:33.640631 18700 raft_consensus.cc:729] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58fc2b24877144eb9476288668215605, State: Initialized, Role: FOLLOWER
I20260806 04:43:33.639804 18699 raft_consensus.cc:374] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:33.641086 18700 consensus_queue.cc:260] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be21577bc87d47c1a385c341db03ac2a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42419 } } peers { permanent_uuid: "77cc566d595d4781b977393a65d44d3e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 36253 } } peers { permanent_uuid: "58fc2b24877144eb9476288668215605" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39793 } }
I20260806 04:43:33.641237 18699 raft_consensus.cc:729] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77cc566d595d4781b977393a65d44d3e, State: Initialized, Role: FOLLOWER
I20260806 04:43:33.641691 18699 consensus_queue.cc:260] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be21577bc87d47c1a385c341db03ac2a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42419 } } peers { permanent_uuid: "77cc566d595d4781b977393a65d44d3e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 36253 } } peers { permanent_uuid: "58fc2b24877144eb9476288668215605" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39793 } }
I20260806 04:43:33.641801 18398 heartbeater.cc:503] Master 127.31.166.190:43183 was elected leader, sending a full tablet report...
I20260806 04:43:33.642148 18700 ts_tablet_manager.cc:1428] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260806 04:43:33.642602 18699 ts_tablet_manager.cc:1428] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:43:33.642673 18679 heartbeater.cc:503] Master 127.31.166.190:43183 was elected leader, sending a full tablet report...
W20260806 04:43:33.805510 18399 tablet.cc:2369] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:43:33.807608 18680 tablet.cc:2369] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:43:33.807708 18536 tablet.cc:2369] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:43:34.089102 18704 raft_consensus.cc:482] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:43:34.089299 18704 raft_consensus.cc:504] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be21577bc87d47c1a385c341db03ac2a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42419 } } peers { permanent_uuid: "77cc566d595d4781b977393a65d44d3e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 36253 } } peers { permanent_uuid: "58fc2b24877144eb9476288668215605" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39793 } }
I20260806 04:43:34.090464 18704 leader_election.cc:288] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77cc566d595d4781b977393a65d44d3e (127.31.166.131:36253), 58fc2b24877144eb9476288668215605 (127.31.166.129:39793)
I20260806 04:43:34.096208 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:34.090876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:33085 (local address 127.31.166.131:36253)
0806 04:43:34.091024 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:34.091032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:34.091299 (+ 267us) server_negotiation.cc:407] Connection header received
0806 04:43:34.091403 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:34.091407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:34.091467 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:34.091560 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:34.091987 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.092779 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.094754 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.095395 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.095547 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:34.095602 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:34.095989 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 04:43:34.096084 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:43:34.096249 18708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:34.090787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:36253 (local address 127.31.166.130:33085)
0806 04:43:34.091158 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:34.091172 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:34.091298 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:34.091586 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:34.091589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:34.091615 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:34.091835 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:34.091843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.092947 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.092950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:34.094588 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:34.094601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.095572 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.095577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:34.095823 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:34.095861 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 04:43:34.095957 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 04:43:34.096058 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":267,"spinlock_wait_cycles":29440,"thread_start_us":96,"threads_started":1}
I20260806 04:43:34.096244 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:34.090919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:41275 (local address 127.31.166.129:39793)
0806 04:43:34.091165 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:34.091169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:34.091299 (+ 130us) server_negotiation.cc:407] Connection header received
0806 04:43:34.091406 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:34.091409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:34.091464 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:34.091566 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:34.091987 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.092819 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.094754 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.095449 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.095558 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:34.095602 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:34.096038 (+ 436us) server_negotiation.cc:299] Negotiation successful
0806 04:43:34.096128 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260806 04:43:34.096652 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:34.090769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:39793 (local address 127.31.166.130:41275)
0806 04:43:34.091133 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:34.091150 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:34.091298 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:34.091586 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:34.091589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:34.091615 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:34.091836 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:34.091843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.092947 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.092950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:34.094606 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:34.094612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.095572 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.095577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:34.095829 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:34.095912 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 04:43:34.096466 (+ 554us) client_negotiation.cc:240] Negotiation successful
0806 04:43:34.096533 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":324,"spinlock_wait_cycles":12416,"thread_start_us":101,"threads_started":1}
I20260806 04:43:34.097347 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1789bbc44f904ca2ba1cb78e2ef76f56" candidate_uuid: "be21577bc87d47c1a385c341db03ac2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77cc566d595d4781b977393a65d44d3e" is_pre_election: true
I20260806 04:43:34.097354 18349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1789bbc44f904ca2ba1cb78e2ef76f56" candidate_uuid: "be21577bc87d47c1a385c341db03ac2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58fc2b24877144eb9476288668215605" is_pre_election: true
I20260806 04:43:34.097671 18629 raft_consensus.cc:2454] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be21577bc87d47c1a385c341db03ac2a in term 0.
I20260806 04:43:34.097676 18349 raft_consensus.cc:2454] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be21577bc87d47c1a385c341db03ac2a in term 0.
I20260806 04:43:34.098208 18420 leader_election.cc:302] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 58fc2b24877144eb9476288668215605, 77cc566d595d4781b977393a65d44d3e, be21577bc87d47c1a385c341db03ac2a; no voters:
I20260806 04:43:34.098489 18704 raft_consensus.cc:2781] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:43:34.098584 18704 raft_consensus.cc:482] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:43:34.098662 18704 raft_consensus.cc:3037] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:34.099826 18704 raft_consensus.cc:504] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be21577bc87d47c1a385c341db03ac2a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42419 } } peers { permanent_uuid: "77cc566d595d4781b977393a65d44d3e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 36253 } } peers { permanent_uuid: "58fc2b24877144eb9476288668215605" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39793 } }
I20260806 04:43:34.100226 18704 leader_election.cc:288] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [CANDIDATE]: Term 1 election: Requested vote from peers 77cc566d595d4781b977393a65d44d3e (127.31.166.131:36253), 58fc2b24877144eb9476288668215605 (127.31.166.129:39793)
I20260806 04:43:34.100625 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1789bbc44f904ca2ba1cb78e2ef76f56" candidate_uuid: "be21577bc87d47c1a385c341db03ac2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77cc566d595d4781b977393a65d44d3e"
I20260806 04:43:34.100636 18349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1789bbc44f904ca2ba1cb78e2ef76f56" candidate_uuid: "be21577bc87d47c1a385c341db03ac2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58fc2b24877144eb9476288668215605"
I20260806 04:43:34.100734 18629 raft_consensus.cc:3037] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:34.100821 18349 raft_consensus.cc:3037] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:34.101958 18349 raft_consensus.cc:2454] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be21577bc87d47c1a385c341db03ac2a in term 1.
I20260806 04:43:34.101997 18629 raft_consensus.cc:2454] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be21577bc87d47c1a385c341db03ac2a in term 1.
I20260806 04:43:34.102344 18421 leader_election.cc:302] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58fc2b24877144eb9476288668215605, be21577bc87d47c1a385c341db03ac2a; no voters:
I20260806 04:43:34.102542 18704 raft_consensus.cc:2781] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:34.102790 18704 raft_consensus.cc:686] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [term 1 LEADER]: Becoming Leader. State: Replica: be21577bc87d47c1a385c341db03ac2a, State: Running, Role: LEADER
I20260806 04:43:34.103183 18704 consensus_queue.cc:237] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be21577bc87d47c1a385c341db03ac2a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42419 } } peers { permanent_uuid: "77cc566d595d4781b977393a65d44d3e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 36253 } } peers { permanent_uuid: "58fc2b24877144eb9476288668215605" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39793 } }
I20260806 04:43:34.106259 18195 catalog_manager.cc:5495] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a reported cstate change: term changed from 0 to 1, leader changed from <none> to be21577bc87d47c1a385c341db03ac2a (127.31.166.130). New cstate: current_term: 1 leader_uuid: "be21577bc87d47c1a385c341db03ac2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be21577bc87d47c1a385c341db03ac2a" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77cc566d595d4781b977393a65d44d3e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58fc2b24877144eb9476288668215605" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39793 } health_report { overall_health: UNKNOWN } } }
I20260806 04:43:34.181872 18714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:34.178321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.31.166.130:42419)
0806 04:43:34.178579 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:34.178582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:34.178596 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:43:34.178707 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:34.178710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:34.178782 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:34.178898 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:43:34.179124 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.179626 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.180485 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.180700 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.180778 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:34.181066 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:43:34.181477 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:43:34.181568 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:34.181687 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 04:43:34.181736 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":105,"threads_started":1}
I20260806 04:43:34.187947 18629 raft_consensus.cc:1261] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e [term 1 FOLLOWER]: Refusing update from remote peer be21577bc87d47c1a385c341db03ac2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:43:34.187928 18349 raft_consensus.cc:1261] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605 [term 1 FOLLOWER]: Refusing update from remote peer be21577bc87d47c1a385c341db03ac2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:43:34.188789 18710 consensus_queue.cc:1035] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58fc2b24877144eb9476288668215605" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:43:34.189062 18704 consensus_queue.cc:1035] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77cc566d595d4781b977393a65d44d3e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:43:34.198295 18720 mvcc.cc:205] Tried to move back new op lower bound from 7315420832505417728 to 7315420832168980480. Current Snapshot: MvccSnapshot[applied={T|T < 7315420832505417728}]
I20260806 04:43:34.200111 18719 mvcc.cc:205] Tried to move back new op lower bound from 7315420832505417728 to 7315420832168980480. Current Snapshot: MvccSnapshot[applied={T|T < 7315420832505417728}]
I20260806 04:43:34.205143 18195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:43:34.205327 18195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:43:34.207808 18195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40db9c16c6a6421888bdfd8a0d317375: Sending DeleteTablet for 3 replicas of tablet 1789bbc44f904ca2ba1cb78e2ef76f56
I20260806 04:43:34.208696 18467 tablet_service.cc:1514] Processing DeleteTablet for tablet 1789bbc44f904ca2ba1cb78e2ef76f56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:34 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:43298
I20260806 04:43:34.208832 18329 tablet_service.cc:1514] Processing DeleteTablet for tablet 1789bbc44f904ca2ba1cb78e2ef76f56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:34 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37170
I20260806 04:43:34.209200 18729 tablet_replica.cc:330] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a: stopping tablet replica
I20260806 04:43:34.209321 18609 tablet_service.cc:1514] Processing DeleteTablet for tablet 1789bbc44f904ca2ba1cb78e2ef76f56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:34 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:59912
I20260806 04:43:34.209635 18729 raft_consensus.cc:2229] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:43:34.209795 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 18263
I20260806 04:43:34.209793 18730 tablet_replica.cc:330] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605: stopping tablet replica
I20260806 04:43:34.210074 18730 raft_consensus.cc:2229] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 58fc2b24877144eb9476288668215605 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:43:34.210245 18729 raft_consensus.cc:2258] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:43:34.211723 18729 ts_tablet_manager.cc:1905] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:43:34.215227 18731 tablet_replica.cc:330] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e: stopping tablet replica
I20260806 04:43:34.215358 18729 ts_tablet_manager.cc:1918] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:43:34.215466 18729 log.cc:1196] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/1789bbc44f904ca2ba1cb78e2ef76f56
I20260806 04:43:34.215695 18731 raft_consensus.cc:2229] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:43:34.215790 18729 ts_tablet_manager.cc:1939] T 1789bbc44f904ca2ba1cb78e2ef76f56 P be21577bc87d47c1a385c341db03ac2a: Deleting consensus metadata
I20260806 04:43:34.216125 18731 raft_consensus.cc:2258] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:43:34.217056 18182 catalog_manager.cc:4841] TS be21577bc87d47c1a385c341db03ac2a (127.31.166.130:42419): tablet 1789bbc44f904ca2ba1cb78e2ef76f56 (table test-table [id=5b7ac29d95c14036aef0a00c869f4b98]) successfully deleted
I20260806 04:43:34.217566 18731 ts_tablet_manager.cc:1905] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:43:34.220201 18731 ts_tablet_manager.cc:1918] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:43:34.220299 18731 log.cc:1196] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/1789bbc44f904ca2ba1cb78e2ef76f56
W20260806 04:43:34.220690 18180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:39793 (error 108)
I20260806 04:43:34.220958 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 18404
I20260806 04:43:34.221881 18731 ts_tablet_manager.cc:1939] T 1789bbc44f904ca2ba1cb78e2ef76f56 P 77cc566d595d4781b977393a65d44d3e: Deleting consensus metadata
I20260806 04:43:34.222579 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:34.221455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:39793 (local address 127.0.0.1:37174)
0806 04:43:34.222376 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:34.222456 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":318,"threads_started":1}
W20260806 04:43:34.222774 18180 catalog_manager.cc:4584] TS 58fc2b24877144eb9476288668215605 (127.31.166.129:39793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1789bbc44f904ca2ba1cb78e2ef76f56: Network error: Client connection negotiation failed: client connection to 127.31.166.129:39793: connect: Connection refused (error 111)
I20260806 04:43:34.223120 18179 catalog_manager.cc:4841] TS 77cc566d595d4781b977393a65d44d3e (127.31.166.131:36253): tablet 1789bbc44f904ca2ba1cb78e2ef76f56 (table test-table [id=5b7ac29d95c14036aef0a00c869f4b98]) successfully deleted
I20260806 04:43:34.228192 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 18542
I20260806 04:43:34.234710 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 18161
2026-08-06T04:43:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2930 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T04:43:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:34Z Disabled control of system clock
I20260806 04:43:34.266363 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:44719
--webserver_interface=127.31.166.190
--webserver_port=0
--builtin_ntp_servers=127.31.166.148:36579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:44719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:43:34.378229 18738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:34.378508 18738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:34.378546 18738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:34.382655 18738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:34.382719 18738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:34.382735 18738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:34.382751 18738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:34.382767 18738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:34.387591 18738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:44719
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:44719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:34.388752 18738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:34.390154 18738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:34.398942 18746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:34.398945 18744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:34.399034 18738 server_base.cc:1034] running on GCE node
W20260806 04:43:34.398950 18743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:34.399902 18738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:34.401118 18738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:34.402401 18738 hybrid_clock.cc:648] HybridClock initialized: now 1785991414402385 us; error 37 us; skew 500 ppm
I20260806 04:43:34.410226 18738 webserver.cc:460] Webserver started at http://127.31.166.190:35449/ using document root <none> and password file <none>
I20260806 04:43:34.410895 18738 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:34.411011 18738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:34.411294 18738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:34.413549 18738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "5a8284166af44b1ab3a74710e2973618"
format_stamp: "Formatted at 2026-08-06 04:43:34 on dist-test-slave-vkjp"
server_key: "d9d10108ae9396d2604d1e72bfb8019c"
server_key_iv: "86d2250149157b641ec110e2dc2ef5d1"
server_key_version: "encryptionkey@0"
I20260806 04:43:34.414150 18738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "5a8284166af44b1ab3a74710e2973618"
format_stamp: "Formatted at 2026-08-06 04:43:34 on dist-test-slave-vkjp"
server_key: "d9d10108ae9396d2604d1e72bfb8019c"
server_key_iv: "86d2250149157b641ec110e2dc2ef5d1"
server_key_version: "encryptionkey@0"
I20260806 04:43:34.418049 18738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 04:43:34.420527 18752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:34.421772 18738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:43:34.421927 18738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "5a8284166af44b1ab3a74710e2973618"
format_stamp: "Formatted at 2026-08-06 04:43:34 on dist-test-slave-vkjp"
server_key: "d9d10108ae9396d2604d1e72bfb8019c"
server_key_iv: "86d2250149157b641ec110e2dc2ef5d1"
server_key_version: "encryptionkey@0"
I20260806 04:43:34.422061 18738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:34.439641 18738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:34.440354 18738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:34.440560 18738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:34.447293 18738 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:44719
I20260806 04:43:34.447332 18806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:44719 every 8 connection(s)
I20260806 04:43:34.448316 18738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:34.451519 18807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:34.452713 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 18738
I20260806 04:43:34.452862 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:34.453182 32410 external_mini_cluster.cc:1428] Setting key f3fb2b2284b9bcf84a67345895922bb6
I20260806 04:43:34.458526 18807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618: Bootstrap starting.
I20260806 04:43:34.460789 18807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:34.461534 18807 log.cc:824] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:34.463672 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:34.454631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.31.166.190:44719)
0806 04:43:34.455336 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:34.455353 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:34.455418 (+ 65us) server_negotiation.cc:407] Connection header received
0806 04:43:34.456221 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:34.456239 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:34.456824 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:34.457182 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:34.457682 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.458739 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.459775 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.460292 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.460482 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:34.461493 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:34.461507 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:34.461521 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:34.462036 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:34.462115 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:34.462147 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:34.462276 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:34.462497 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 04:43:34.462860 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":102,"threads_started":1}
I20260806 04:43:34.463750 18807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618: No bootstrap required, opened a new log
I20260806 04:43:34.467555 18807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8284166af44b1ab3a74710e2973618" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44719 } }
I20260806 04:43:34.467803 18807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:34.467885 18807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a8284166af44b1ab3a74710e2973618, State: Initialized, Role: FOLLOWER
I20260806 04:43:34.468359 18807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8284166af44b1ab3a74710e2973618" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44719 } }
I20260806 04:43:34.468525 18807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:34.468613 18807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:34.468715 18807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:34.469756 18807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8284166af44b1ab3a74710e2973618" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44719 } }
I20260806 04:43:34.470103 18807 leader_election.cc:302] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a8284166af44b1ab3a74710e2973618; no voters:
I20260806 04:43:34.470431 18807 leader_election.cc:288] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:34.470618 18813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:34.470880 18813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [term 1 LEADER]: Becoming Leader. State: Replica: 5a8284166af44b1ab3a74710e2973618, State: Running, Role: LEADER
I20260806 04:43:34.471232 18813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8284166af44b1ab3a74710e2973618" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44719 } }
I20260806 04:43:34.471714 18807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:34.472972 18815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a8284166af44b1ab3a74710e2973618. Latest consensus state: current_term: 1 leader_uuid: "5a8284166af44b1ab3a74710e2973618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8284166af44b1ab3a74710e2973618" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44719 } } }
I20260806 04:43:34.473110 18815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:34.473385 18814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a8284166af44b1ab3a74710e2973618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a8284166af44b1ab3a74710e2973618" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 44719 } } }
I20260806 04:43:34.473582 18814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618 [sys.catalog]: This master's current role is: LEADER
W20260806 04:43:34.476963 18828 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:43:34.477038 18828 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:43:34.477128 18822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:34.477931 18822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:34.483923 18822 catalog_manager.cc:1347] Generated new cluster ID: 82a5516f35de45cbbc48994d928064b2
I20260806 04:43:34.484012 18822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:34.494971 18822 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:34.496182 18822 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:34.513002 18822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618: Generated new TSK 0
I20260806 04:43:34.513984 18822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:43:34.525098 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:44719
--builtin_ntp_servers=127.31.166.148:36579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:43:34.648502 18833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:34.648845 18833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:34.648937 18833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:34.653548 18833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:34.653637 18833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:34.653738 18833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:43:34.659523 18833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:44719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:34.660682 18833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:34.662465 18833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:34.672784 18841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:34.672869 18838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:34.672822 18839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:34.673157 18833 server_base.cc:1034] running on GCE node
I20260806 04:43:34.673575 18833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:34.674288 18833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:34.675444 18833 hybrid_clock.cc:648] HybridClock initialized: now 1785991414675418 us; error 38 us; skew 500 ppm
I20260806 04:43:34.677366 18833 webserver.cc:460] Webserver started at http://127.31.166.129:38327/ using document root <none> and password file <none>
I20260806 04:43:34.677946 18833 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:34.678007 18833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:34.678223 18833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:34.680349 18833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "d600edb34ee045cda33de93056f3bf39"
format_stamp: "Formatted at 2026-08-06 04:43:34 on dist-test-slave-vkjp"
server_key: "f31fc9c9133c939967786add2dda384f"
server_key_iv: "5740bf80aca4d620a213e2663e8d6f67"
server_key_version: "encryptionkey@0"
I20260806 04:43:34.680913 18833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "d600edb34ee045cda33de93056f3bf39"
format_stamp: "Formatted at 2026-08-06 04:43:34 on dist-test-slave-vkjp"
server_key: "f31fc9c9133c939967786add2dda384f"
server_key_iv: "5740bf80aca4d620a213e2663e8d6f67"
server_key_version: "encryptionkey@0"
I20260806 04:43:34.684625 18833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:43:34.686986 18847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:34.688221 18833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:43:34.688351 18833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "d600edb34ee045cda33de93056f3bf39"
format_stamp: "Formatted at 2026-08-06 04:43:34 on dist-test-slave-vkjp"
server_key: "f31fc9c9133c939967786add2dda384f"
server_key_iv: "5740bf80aca4d620a213e2663e8d6f67"
server_key_version: "encryptionkey@0"
I20260806 04:43:34.688473 18833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:34.711145 18833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:34.711882 18833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:34.712092 18833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:34.712754 18833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:34.713732 18833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:34.713809 18833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:34.713886 18833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:34.713936 18833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:34.723531 18833 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:41441
I20260806 04:43:34.723557 18969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:41441 every 8 connection(s)
I20260806 04:43:34.724491 18833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:43:34.732822 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 18833
I20260806 04:43:34.732967 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:43:34.733245 32410 external_mini_cluster.cc:1428] Setting key d935e3e33916b9b34d5240f707f01265
I20260806 04:43:34.733946 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:34.726285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:51861 (local address 127.31.166.190:44719)
0806 04:43:34.726437 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:34.726441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:34.727667 (+ 1226us) server_negotiation.cc:407] Connection header received
0806 04:43:34.729029 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:34.729032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:34.729081 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:34.729158 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:34.730077 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.730643 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.731851 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.732073 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.732156 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:34.733418 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:34.733422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:34.733425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:34.733438 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:34.733461 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:34.733471 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:34.733551 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:34.733717 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 04:43:34.733782 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:43:34.734929 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:34.726729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:44719 (local address 127.31.166.129:51861)
0806 04:43:34.727512 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:34.727556 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:34.728823 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:34.729353 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:34.729359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:34.729441 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:34.729924 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:34.729937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.730760 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.730763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:34.731733 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:34.731743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.732206 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.732208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:34.732392 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:34.733130 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:43:34.733132 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:34.733171 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:43:34.733230 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:34.733561 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:34.733569 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:34.733573 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:43:34.733703 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 04:43:34.733969 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":110,"threads_started":1}
I20260806 04:43:34.735414 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:44719
--builtin_ntp_servers=127.31.166.148:36579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:43:34.736322 18970 heartbeater.cc:347] Connected to a master server at 127.31.166.190:44719
I20260806 04:43:34.736570 18970 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:34.737080 18970 heartbeater.cc:511] Master 127.31.166.190:44719 requested a full tablet report, sending...
I20260806 04:43:34.738669 18769 ts_manager.cc:194] Registered new tserver with Master: d600edb34ee045cda33de93056f3bf39 (127.31.166.129:41441)
I20260806 04:43:34.740417 18769 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.166.129:51861
W20260806 04:43:34.847162 18973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:34.847430 18973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:34.847468 18973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:34.851521 18973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:34.851572 18973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:34.851624 18973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:43:34.857185 18973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:44719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:34.858294 18973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:34.859687 18973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:34.869375 18978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:34.869378 18982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:34.869375 18979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:34.869553 18973 server_base.cc:1034] running on GCE node
I20260806 04:43:34.870049 18973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:34.870673 18973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:34.871867 18973 hybrid_clock.cc:648] HybridClock initialized: now 1785991414871839 us; error 27 us; skew 500 ppm
I20260806 04:43:34.874019 18973 webserver.cc:460] Webserver started at http://127.31.166.130:46509/ using document root <none> and password file <none>
I20260806 04:43:34.874598 18973 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:34.874660 18973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:34.874877 18973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:34.876951 18973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "6d0aeeafe52349dca277bc911aeb79d5"
format_stamp: "Formatted at 2026-08-06 04:43:34 on dist-test-slave-vkjp"
server_key: "b6432a641616a3d006335eb269c3e65c"
server_key_iv: "197974c739ba26a522ee36c2ffd3f6d9"
server_key_version: "encryptionkey@0"
I20260806 04:43:34.877503 18973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "6d0aeeafe52349dca277bc911aeb79d5"
format_stamp: "Formatted at 2026-08-06 04:43:34 on dist-test-slave-vkjp"
server_key: "b6432a641616a3d006335eb269c3e65c"
server_key_iv: "197974c739ba26a522ee36c2ffd3f6d9"
server_key_version: "encryptionkey@0"
I20260806 04:43:34.881335 18973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:43:34.883862 18988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:34.885056 18973 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:43:34.885195 18973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "6d0aeeafe52349dca277bc911aeb79d5"
format_stamp: "Formatted at 2026-08-06 04:43:34 on dist-test-slave-vkjp"
server_key: "b6432a641616a3d006335eb269c3e65c"
server_key_iv: "197974c739ba26a522ee36c2ffd3f6d9"
server_key_version: "encryptionkey@0"
I20260806 04:43:34.885303 18973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:34.899025 18973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:34.899784 18973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:34.899981 18973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:34.900576 18973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:34.901510 18973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:34.901556 18973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:34.901623 18973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:34.901660 18973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:34.910622 18973 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:43871
I20260806 04:43:34.910670 19110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:43871 every 8 connection(s)
I20260806 04:43:34.911684 18973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:43:34.912561 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 18973
I20260806 04:43:34.912710 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:43:34.912993 32410 external_mini_cluster.cc:1428] Setting key 9c69004e3c3c89fa2c19749843e9cc76
I20260806 04:43:34.915222 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:44719
--builtin_ntp_servers=127.31.166.148:36579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:43:34.923069 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:34.913649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:39265 (local address 127.31.166.190:44719)
0806 04:43:34.913808 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:34.913813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:34.915067 (+ 1254us) server_negotiation.cc:407] Connection header received
0806 04:43:34.917208 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:34.917213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:34.917286 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:34.917403 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:34.918369 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.919311 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.920620 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.921079 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.921231 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:34.922418 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:34.922426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:34.922431 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:34.922451 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:34.922480 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:34.922496 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:34.922599 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:34.922736 (+ 137us) server_negotiation.cc:299] Negotiation successful
0806 04:43:34.922835 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:43:34.923869 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:34.914080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:44719 (local address 127.31.166.130:39265)
0806 04:43:34.914878 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:34.914938 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:34.916960 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:34.917577 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:34.917584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:34.917674 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:34.918198 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:34.918211 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.919464 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.919467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:34.920477 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:34.920487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:34.921254 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:34.921257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:34.921434 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:34.922102 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:43:34.922103 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:34.922146 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:43:34.922223 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:34.922594 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:34.922600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:34.922604 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:43:34.922695 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 04:43:34.922937 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":174,"threads_started":1}
I20260806 04:43:34.925176 19111 heartbeater.cc:347] Connected to a master server at 127.31.166.190:44719
I20260806 04:43:34.925505 19111 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:34.926295 19111 heartbeater.cc:511] Master 127.31.166.190:44719 requested a full tablet report, sending...
I20260806 04:43:34.927726 18769 ts_manager.cc:194] Registered new tserver with Master: 6d0aeeafe52349dca277bc911aeb79d5 (127.31.166.130:43871)
I20260806 04:43:34.928507 18769 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.166.130:39265
W20260806 04:43:35.033183 19114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:35.033486 19114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:35.033558 19114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:35.037618 19114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:35.037703 19114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:35.037796 19114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:43:35.042503 19114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:44719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:35.043682 19114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:35.045068 19114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:43:35.054365 19114 server_base.cc:1034] running on GCE node
W20260806 04:43:35.054380 19119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:35.054415 19122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:35.054334 19120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:35.055073 19114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:35.055856 19114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:35.057019 19114 hybrid_clock.cc:648] HybridClock initialized: now 1785991415057006 us; error 23 us; skew 500 ppm
I20260806 04:43:35.058821 19114 webserver.cc:460] Webserver started at http://127.31.166.131:42745/ using document root <none> and password file <none>
I20260806 04:43:35.059413 19114 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:35.059478 19114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:35.059643 19114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:35.061604 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "7bdf148c61fb4cf6b02a17d6db574d31"
format_stamp: "Formatted at 2026-08-06 04:43:35 on dist-test-slave-vkjp"
server_key: "aa76672b26dbf2d6e9565d058c563e0c"
server_key_iv: "de6bba4cb3c488815105fa12a829e79f"
server_key_version: "encryptionkey@0"
I20260806 04:43:35.062179 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "7bdf148c61fb4cf6b02a17d6db574d31"
format_stamp: "Formatted at 2026-08-06 04:43:35 on dist-test-slave-vkjp"
server_key: "aa76672b26dbf2d6e9565d058c563e0c"
server_key_iv: "de6bba4cb3c488815105fa12a829e79f"
server_key_version: "encryptionkey@0"
I20260806 04:43:35.065886 19114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260806 04:43:35.068238 19129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:35.069437 19114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:43:35.069525 19114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "7bdf148c61fb4cf6b02a17d6db574d31"
format_stamp: "Formatted at 2026-08-06 04:43:35 on dist-test-slave-vkjp"
server_key: "aa76672b26dbf2d6e9565d058c563e0c"
server_key_iv: "de6bba4cb3c488815105fa12a829e79f"
server_key_version: "encryptionkey@0"
I20260806 04:43:35.069604 19114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:35.085824 19114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:35.086457 19114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:35.086607 19114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:35.087168 19114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:35.088109 19114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:35.088157 19114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:35.088196 19114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:35.088212 19114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:35.097980 19114 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:43089
I20260806 04:43:35.098009 19248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:43089 every 8 connection(s)
I20260806 04:43:35.098909 19114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:43:35.102753 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 19114
I20260806 04:43:35.102849 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:43:35.103178 32410 external_mini_cluster.cc:1428] Setting key 805c4d010cf1d8fcc37c772fa67c1426
I20260806 04:43:35.108808 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.100929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:38095 (local address 127.31.166.190:44719)
0806 04:43:35.101103 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:35.101107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:35.102124 (+ 1017us) server_negotiation.cc:407] Connection header received
0806 04:43:35.103709 (+ 1585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:35.103712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:35.103765 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:35.103877 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:35.105042 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.105713 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.106828 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.107044 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.107218 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:35.108403 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:35.108405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:35.108407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:35.108418 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:35.108436 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:35.108444 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:35.108504 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:35.108646 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 04:43:35.108695 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 04:43:35.109691 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.101330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:44719 (local address 127.31.166.131:38095)
0806 04:43:35.101976 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:35.102016 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:35.103444 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:35.104075 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:35.104085 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:35.104191 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:35.104900 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:35.104920 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.105835 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.105838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:35.106712 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:35.106722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.107278 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.107283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:35.107436 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:35.108157 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:43:35.108159 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:35.108198 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:43:35.108263 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:35.108524 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:35.108531 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:35.108535 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:43:35.108643 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 04:43:35.108916 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":118,"threads_started":1}
I20260806 04:43:35.110932 19249 heartbeater.cc:347] Connected to a master server at 127.31.166.190:44719
I20260806 04:43:35.111212 19249 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:35.111760 19249 heartbeater.cc:511] Master 127.31.166.190:44719 requested a full tablet report, sending...
I20260806 04:43:35.112767 18769 ts_manager.cc:194] Registered new tserver with Master: 7bdf148c61fb4cf6b02a17d6db574d31 (127.31.166.131:43089)
I20260806 04:43:35.113324 18769 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.166.131:38095
I20260806 04:43:35.117070 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:43:35.122754 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.119205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.31.166.190:44719)
0806 04:43:35.119339 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:35.119352 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:35.119493 (+ 141us) server_negotiation.cc:407] Connection header received
0806 04:43:35.119613 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:35.119615 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:35.119653 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:35.119709 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:35.119929 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.120405 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.121224 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.121454 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.121529 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:35.122336 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:35.122338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:35.122340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:35.122378 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:35.122394 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:35.122401 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:35.122456 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:35.122539 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 04:43:35.122581 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:43:35.125725 18769 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:43:35.128880 18769 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:43:35.142777 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.137288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:43871 (local address 127.0.0.1:39260)
0806 04:43:35.137667 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:35.137723 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:35.137855 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:35.138684 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:35.138689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:35.138711 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:35.138846 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:35.138856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.140131 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.140134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:35.141421 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:35.141430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.142275 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.142279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:35.142387 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:35.142428 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 04:43:35.142511 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 04:43:35.142621 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":99,"threads_started":1}
I20260806 04:43:35.143414 19264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.137684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:41441 (local address 127.0.0.1:48908)
0806 04:43:35.138221 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:35.138234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:35.138325 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:35.139430 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:35.139433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:35.139452 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:35.139536 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:35.139542 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.140911 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.140914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:35.141947 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:35.141956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.143029 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.143033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:35.143186 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:35.143214 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 04:43:35.143257 (+ 43us) client_negotiation.cc:240] Negotiation successful
0806 04:43:35.143306 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":21,"thread_start_us":84,"threads_started":1}
I20260806 04:43:35.143669 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.137407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39260 (local address 127.31.166.130:43871)
0806 04:43:35.138121 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:35.138136 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:35.138161 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:43:35.138242 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:35.138251 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:35.138488 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:35.138690 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:35.138982 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.140020 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.141532 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.142135 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.142845 (+ 710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:35.143343 (+ 498us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:35.143394 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 04:43:35.143499 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":124,"threads_started":1}
I20260806 04:43:35.144090 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.138500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48908 (local address 127.31.166.129:41441)
0806 04:43:35.138958 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:35.138974 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:35.138998 (+ 24us) server_negotiation.cc:407] Connection header received
0806 04:43:35.139074 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:35.139105 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:35.139287 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:35.139435 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:35.139670 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.140765 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.142074 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.142867 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.143645 (+ 778us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:35.143826 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:35.143875 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 04:43:35.143965 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":152,"threads_started":1}
I20260806 04:43:35.144780 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.138003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:43089 (local address 127.0.0.1:47388)
0806 04:43:35.138926 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:35.138940 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:35.139040 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:35.139732 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:35.139735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:35.139753 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:35.139842 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:35.139849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.140879 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.140882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:35.142643 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:35.142650 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.144149 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.144382 (+ 233us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:35.144480 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:35.144507 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 04:43:35.144590 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 04:43:35.144657 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"mutex_wait_us":36,"thread_start_us":97,"threads_started":1}
I20260806 04:43:35.144905 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.138162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47388 (local address 127.31.166.131:43089)
0806 04:43:35.138583 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:35.138598 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:35.139173 (+ 575us) server_negotiation.cc:407] Connection header received
0806 04:43:35.139245 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:35.139249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:35.139421 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:35.139549 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:35.139972 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.140741 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.142759 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.143792 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.143916 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:35.144083 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:35.144636 (+ 553us) server_negotiation.cc:299] Negotiation successful
0806 04:43:35.144737 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":123,"threads_started":1}
I20260806 04:43:35.146528 19040 tablet_service.cc:1467] Processing CreateTablet for tablet 18e0e37fe2ca4d2596761ba06aa127fc (DEFAULT_TABLE table=test-table [id=990efa20551c4bb181b6f86d4901c9e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:35.146914 18900 tablet_service.cc:1467] Processing CreateTablet for tablet 18e0e37fe2ca4d2596761ba06aa127fc (DEFAULT_TABLE table=test-table [id=990efa20551c4bb181b6f86d4901c9e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:35.147497 19040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e0e37fe2ca4d2596761ba06aa127fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:35.147320 19180 tablet_service.cc:1467] Processing CreateTablet for tablet 18e0e37fe2ca4d2596761ba06aa127fc (DEFAULT_TABLE table=test-table [id=990efa20551c4bb181b6f86d4901c9e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:35.147925 18900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e0e37fe2ca4d2596761ba06aa127fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:35.148219 19180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e0e37fe2ca4d2596761ba06aa127fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:35.153689 19268 tablet_bootstrap.cc:492] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5: Bootstrap starting.
I20260806 04:43:35.155393 19269 tablet_bootstrap.cc:492] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31: Bootstrap starting.
I20260806 04:43:35.156358 19270 tablet_bootstrap.cc:492] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39: Bootstrap starting.
I20260806 04:43:35.156924 19268 tablet_bootstrap.cc:654] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:35.157563 19269 tablet_bootstrap.cc:654] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:35.157867 19268 log.cc:824] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:35.158372 19269 log.cc:824] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:35.158583 19270 tablet_bootstrap.cc:654] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:35.159389 19270 log.cc:824] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:35.160020 19268 tablet_bootstrap.cc:492] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5: No bootstrap required, opened a new log
I20260806 04:43:35.160228 19268 ts_tablet_manager.cc:1397] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 04:43:35.160640 19269 tablet_bootstrap.cc:492] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31: No bootstrap required, opened a new log
I20260806 04:43:35.160827 19269 ts_tablet_manager.cc:1397] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:43:35.161407 19270 tablet_bootstrap.cc:492] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39: No bootstrap required, opened a new log
I20260806 04:43:35.161600 19270 ts_tablet_manager.cc:1397] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:43:35.164713 19270 raft_consensus.cc:348] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0aeeafe52349dca277bc911aeb79d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 43871 } } peers { permanent_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 43089 } } peers { permanent_uuid: "d600edb34ee045cda33de93056f3bf39" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 41441 } }
I20260806 04:43:35.164958 19270 raft_consensus.cc:374] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:35.164747 19268 raft_consensus.cc:348] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0aeeafe52349dca277bc911aeb79d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 43871 } } peers { permanent_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 43089 } } peers { permanent_uuid: "d600edb34ee045cda33de93056f3bf39" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 41441 } }
I20260806 04:43:35.165010 19270 raft_consensus.cc:729] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d600edb34ee045cda33de93056f3bf39, State: Initialized, Role: FOLLOWER
I20260806 04:43:35.165028 19268 raft_consensus.cc:374] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:35.165084 19268 raft_consensus.cc:729] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d0aeeafe52349dca277bc911aeb79d5, State: Initialized, Role: FOLLOWER
I20260806 04:43:35.164985 19269 raft_consensus.cc:348] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0aeeafe52349dca277bc911aeb79d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 43871 } } peers { permanent_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 43089 } } peers { permanent_uuid: "d600edb34ee045cda33de93056f3bf39" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 41441 } }
I20260806 04:43:35.165220 19269 raft_consensus.cc:374] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:35.165273 19269 raft_consensus.cc:729] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bdf148c61fb4cf6b02a17d6db574d31, State: Initialized, Role: FOLLOWER
I20260806 04:43:35.165525 19270 consensus_queue.cc:260] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0aeeafe52349dca277bc911aeb79d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 43871 } } peers { permanent_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 43089 } } peers { permanent_uuid: "d600edb34ee045cda33de93056f3bf39" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 41441 } }
I20260806 04:43:35.165544 19268 consensus_queue.cc:260] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0aeeafe52349dca277bc911aeb79d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 43871 } } peers { permanent_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 43089 } } peers { permanent_uuid: "d600edb34ee045cda33de93056f3bf39" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 41441 } }
I20260806 04:43:35.165622 19269 consensus_queue.cc:260] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0aeeafe52349dca277bc911aeb79d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 43871 } } peers { permanent_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 43089 } } peers { permanent_uuid: "d600edb34ee045cda33de93056f3bf39" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 41441 } }
I20260806 04:43:35.166206 18970 heartbeater.cc:503] Master 127.31.166.190:44719 was elected leader, sending a full tablet report...
I20260806 04:43:35.166453 19111 heartbeater.cc:503] Master 127.31.166.190:44719 was elected leader, sending a full tablet report...
I20260806 04:43:35.166560 19270 ts_tablet_manager.cc:1428] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 04:43:35.167572 19268 ts_tablet_manager.cc:1428] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 04:43:35.168627 19269 ts_tablet_manager.cc:1428] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20260806 04:43:35.168948 19249 heartbeater.cc:503] Master 127.31.166.190:44719 was elected leader, sending a full tablet report...
W20260806 04:43:35.225732 18971 tablet.cc:2369] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:43:35.230299 19276 raft_consensus.cc:482] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:43:35.230631 19276 raft_consensus.cc:504] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0aeeafe52349dca277bc911aeb79d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 43871 } } peers { permanent_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 43089 } } peers { permanent_uuid: "d600edb34ee045cda33de93056f3bf39" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 41441 } }
I20260806 04:43:35.231949 19276 leader_election.cc:288] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d0aeeafe52349dca277bc911aeb79d5 (127.31.166.130:43871), d600edb34ee045cda33de93056f3bf39 (127.31.166.129:41441)
I20260806 04:43:35.236356 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.232299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:37277 (local address 127.31.166.130:43871)
0806 04:43:35.232453 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:35.232459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:35.232477 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:43:35.232539 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:35.232542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:35.232607 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:35.232716 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:35.233040 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.233566 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.235124 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.235742 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.235877 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:35.235931 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:35.236129 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 04:43:35.236221 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:43:35.236377 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.232157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:43871 (local address 127.31.166.131:37277)
0806 04:43:35.232303 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:35.232321 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:35.232424 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:35.232741 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:35.232744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:35.232789 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:35.232918 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:35.232925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.233682 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.233687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:35.234960 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:35.234970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.235875 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.235877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:35.235986 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:35.236017 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 04:43:35.236098 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 04:43:35.236171 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":27,"spinlock_wait_cycles":7168}
I20260806 04:43:35.236451 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.232502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:37689 (local address 127.31.166.129:41441)
0806 04:43:35.232638 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:35.232644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:35.232736 (+ 92us) server_negotiation.cc:407] Connection header received
0806 04:43:35.232910 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:35.232913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:35.232966 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:35.233055 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:35.233375 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.233888 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.235266 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.235899 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.236017 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:35.236048 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:35.236245 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 04:43:35.236315 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:43:35.236392 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.232326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:41441 (local address 127.31.166.131:37689)
0806 04:43:35.232638 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:35.232651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:35.232798 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:35.233065 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:35.233067 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:35.233083 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:35.233231 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:35.233237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.234013 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.234017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:35.235167 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:35.235174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.236022 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.236026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:35.236126 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:35.236145 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:43:35.236211 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:43:35.236266 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":21,"spinlock_wait_cycles":29312,"thread_start_us":143,"threads_started":1}
I20260806 04:43:35.237130 19061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e0e37fe2ca4d2596761ba06aa127fc" candidate_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d0aeeafe52349dca277bc911aeb79d5" is_pre_election: true
I20260806 04:43:35.237246 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e0e37fe2ca4d2596761ba06aa127fc" candidate_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d600edb34ee045cda33de93056f3bf39" is_pre_election: true
I20260806 04:43:35.237437 19061 raft_consensus.cc:2454] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bdf148c61fb4cf6b02a17d6db574d31 in term 0.
I20260806 04:43:35.237563 18921 raft_consensus.cc:2454] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bdf148c61fb4cf6b02a17d6db574d31 in term 0.
I20260806 04:43:35.237989 19131 leader_election.cc:302] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d0aeeafe52349dca277bc911aeb79d5, 7bdf148c61fb4cf6b02a17d6db574d31; no voters:
I20260806 04:43:35.238293 19276 raft_consensus.cc:2781] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:43:35.238396 19276 raft_consensus.cc:482] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:43:35.238456 19276 raft_consensus.cc:3037] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:35.239610 19276 raft_consensus.cc:504] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0aeeafe52349dca277bc911aeb79d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 43871 } } peers { permanent_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 43089 } } peers { permanent_uuid: "d600edb34ee045cda33de93056f3bf39" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 41441 } }
I20260806 04:43:35.239964 19276 leader_election.cc:288] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [CANDIDATE]: Term 1 election: Requested vote from peers 6d0aeeafe52349dca277bc911aeb79d5 (127.31.166.130:43871), d600edb34ee045cda33de93056f3bf39 (127.31.166.129:41441)
I20260806 04:43:35.240324 19061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e0e37fe2ca4d2596761ba06aa127fc" candidate_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d0aeeafe52349dca277bc911aeb79d5"
I20260806 04:43:35.240329 18921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e0e37fe2ca4d2596761ba06aa127fc" candidate_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d600edb34ee045cda33de93056f3bf39"
I20260806 04:43:35.240425 19061 raft_consensus.cc:3037] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:35.240481 18921 raft_consensus.cc:3037] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:35.241483 19061 raft_consensus.cc:2454] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bdf148c61fb4cf6b02a17d6db574d31 in term 1.
I20260806 04:43:35.241505 18921 raft_consensus.cc:2454] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bdf148c61fb4cf6b02a17d6db574d31 in term 1.
I20260806 04:43:35.241823 19131 leader_election.cc:302] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d0aeeafe52349dca277bc911aeb79d5, 7bdf148c61fb4cf6b02a17d6db574d31; no voters:
I20260806 04:43:35.242026 19276 raft_consensus.cc:2781] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:35.242223 19276 raft_consensus.cc:686] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [term 1 LEADER]: Becoming Leader. State: Replica: 7bdf148c61fb4cf6b02a17d6db574d31, State: Running, Role: LEADER
I20260806 04:43:35.242544 19276 consensus_queue.cc:237] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0aeeafe52349dca277bc911aeb79d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 43871 } } peers { permanent_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 43089 } } peers { permanent_uuid: "d600edb34ee045cda33de93056f3bf39" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 41441 } }
I20260806 04:43:35.245749 18769 catalog_manager.cc:5495] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bdf148c61fb4cf6b02a17d6db574d31 (127.31.166.131). New cstate: current_term: 1 leader_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d0aeeafe52349dca277bc911aeb79d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 43871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bdf148c61fb4cf6b02a17d6db574d31" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 43089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d600edb34ee045cda33de93056f3bf39" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 41441 } health_report { overall_health: UNKNOWN } } }
I20260806 04:43:35.267544 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.263507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.31.166.131:43089)
0806 04:43:35.263671 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:35.263676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:35.263693 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:43:35.263882 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:35.263886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:35.263958 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:35.264084 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:43:35.264361 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.264899 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.265837 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:35.266099 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:35.266224 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:35.266566 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:43:35.267022 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:43:35.267213 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:35.267310 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 04:43:35.267370 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:43:35.274076 19061 raft_consensus.cc:1261] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5 [term 1 FOLLOWER]: Refusing update from remote peer 7bdf148c61fb4cf6b02a17d6db574d31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:43:35.274076 18921 raft_consensus.cc:1261] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39 [term 1 FOLLOWER]: Refusing update from remote peer 7bdf148c61fb4cf6b02a17d6db574d31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:43:35.274901 19276 consensus_queue.cc:1035] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d600edb34ee045cda33de93056f3bf39" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 41441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:43:35.275187 19279 consensus_queue.cc:1035] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d0aeeafe52349dca277bc911aeb79d5" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 43871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:43:35.283636 19284 mvcc.cc:205] Tried to move back new op lower bound from 7315420836954279936 to 7315420836836200448. Current Snapshot: MvccSnapshot[applied={T|T < 7315420836954279936}]
I20260806 04:43:35.291203 18767 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:43:35.291400 18767 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:43:35.294063 18767 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a8284166af44b1ab3a74710e2973618: Sending DeleteTablet for 3 replicas of tablet 18e0e37fe2ca4d2596761ba06aa127fc
I20260806 04:43:35.294932 19040 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e0e37fe2ca4d2596761ba06aa127fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:35 UTC) from {username='slave'} at 127.0.0.1:39260
I20260806 04:43:35.294940 18900 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e0e37fe2ca4d2596761ba06aa127fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:35 UTC) from {username='slave'} at 127.0.0.1:48908
I20260806 04:43:35.295513 19294 tablet_replica.cc:330] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39: stopping tablet replica
I20260806 04:43:35.295890 19294 raft_consensus.cc:2229] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:43:35.296052 19180 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e0e37fe2ca4d2596761ba06aa127fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:35 UTC) from {username='slave'} at 127.0.0.1:47388
I20260806 04:43:35.296329 19294 raft_consensus.cc:2258] T 18e0e37fe2ca4d2596761ba06aa127fc P d600edb34ee045cda33de93056f3bf39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:43:35.296532 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 18833
I20260806 04:43:35.296743 19293 tablet_replica.cc:330] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5: stopping tablet replica
I20260806 04:43:35.297107 19293 raft_consensus.cc:2229] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:43:35.297398 19293 raft_consensus.cc:2258] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:43:35.299571 19293 ts_tablet_manager.cc:1905] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:43:35.301903 19293 ts_tablet_manager.cc:1918] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:43:35.302034 19293 log.cc:1196] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/18e0e37fe2ca4d2596761ba06aa127fc
I20260806 04:43:35.302456 19293 ts_tablet_manager.cc:1939] T 18e0e37fe2ca4d2596761ba06aa127fc P 6d0aeeafe52349dca277bc911aeb79d5: Deleting consensus metadata
I20260806 04:43:35.304095 18754 catalog_manager.cc:4841] TS 6d0aeeafe52349dca277bc911aeb79d5 (127.31.166.130:43871): tablet 18e0e37fe2ca4d2596761ba06aa127fc (table test-table [id=990efa20551c4bb181b6f86d4901c9e4]) successfully deleted
I20260806 04:43:35.304737 19295 tablet_replica.cc:330] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31: stopping tablet replica
I20260806 04:43:35.305051 19295 raft_consensus.cc:2229] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:43:35.305393 19295 raft_consensus.cc:2258] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260806 04:43:35.306499 18753 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:41441 (error 108)
I20260806 04:43:35.306602 19295 ts_tablet_manager.cc:1905] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:43:35.307353 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 18973
I20260806 04:43:35.307663 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:35.307298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:41441 (local address 127.0.0.1:48918)
0806 04:43:35.307498 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:35.307573 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:41441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260806 04:43:35.308997 18753 catalog_manager.cc:4584] TS d600edb34ee045cda33de93056f3bf39 (127.31.166.129:41441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 18e0e37fe2ca4d2596761ba06aa127fc: Network error: Client connection negotiation failed: client connection to 127.31.166.129:41441: connect: Connection refused (error 111)
I20260806 04:43:35.309949 19295 ts_tablet_manager.cc:1918] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:43:35.310065 19295 log.cc:1196] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/18e0e37fe2ca4d2596761ba06aa127fc
I20260806 04:43:35.310360 19295 ts_tablet_manager.cc:1939] T 18e0e37fe2ca4d2596761ba06aa127fc P 7bdf148c61fb4cf6b02a17d6db574d31: Deleting consensus metadata
I20260806 04:43:35.315340 18754 catalog_manager.cc:4841] TS 7bdf148c61fb4cf6b02a17d6db574d31 (127.31.166.131:43089): tablet 18e0e37fe2ca4d2596761ba06aa127fc (table test-table [id=990efa20551c4bb181b6f86d4901c9e4]) successfully deleted
I20260806 04:43:35.316481 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 19114
I20260806 04:43:35.322968 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 18738
2026-08-06T04:43:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1087 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:35 dist-test-slave-vkjp krb5kdc[19299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:35 dist-test-slave-vkjp krb5kdc[19299](info): set up 2 sockets
Aug 06 04:43:35 dist-test-slave-vkjp krb5kdc[19299](info): commencing operation
krb5kdc: starting...
W20260806 04:43:36.465389 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:43:36 dist-test-slave-vkjp krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:43:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:43:36.588265 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:45733
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:45733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:36.708932 19314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:36.709282 19314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:36.709363 19314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:36.713928 19314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:36.714046 19314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:36.714090 19314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:36.714131 19314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:36.714171 19314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:36.720131 19314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:45733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:45733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:36.721386 19314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:36.722518 19314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:36.731974 19321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:36.732239 19314 server_base.cc:1034] running on GCE node
W20260806 04:43:36.731978 19320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:36.731974 19323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:36.733351 19314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:36.734526 19314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:36.735720 19314 hybrid_clock.cc:648] HybridClock initialized: now 1785991416735697 us; error 35 us; skew 500 ppm
Aug 06 04:43:36 dist-test-slave-vkjp krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991416, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:36.738636 19314 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:43:36.739683 19314 webserver.cc:460] Webserver started at http://127.31.166.190:34183/ using document root <none> and password file <none>
I20260806 04:43:36.740263 19314 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:36.740360 19314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:36.740602 19314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:36.742658 19314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "0b01d165cd5041a6a16bbaf6d9add678"
format_stamp: "Formatted at 2026-08-06 04:43:36 on dist-test-slave-vkjp"
server_key: "444aaed1af6c04a6d710009f41e2fbd3"
server_key_iv: "1059b655658e4c03c704b5c1d6407705"
server_key_version: "encryptionkey@0"
I20260806 04:43:36.743304 19314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "0b01d165cd5041a6a16bbaf6d9add678"
format_stamp: "Formatted at 2026-08-06 04:43:36 on dist-test-slave-vkjp"
server_key: "444aaed1af6c04a6d710009f41e2fbd3"
server_key_iv: "1059b655658e4c03c704b5c1d6407705"
server_key_version: "encryptionkey@0"
I20260806 04:43:36.747030 19314 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:43:36.749429 19330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:36.750581 19314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:43:36.750728 19314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "0b01d165cd5041a6a16bbaf6d9add678"
format_stamp: "Formatted at 2026-08-06 04:43:36 on dist-test-slave-vkjp"
server_key: "444aaed1af6c04a6d710009f41e2fbd3"
server_key_iv: "1059b655658e4c03c704b5c1d6407705"
server_key_version: "encryptionkey@0"
I20260806 04:43:36.750866 19314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:36.766041 19314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:36.769356 19314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:36.769562 19314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:36.776829 19314 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:45733
I20260806 04:43:36.776840 19387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:45733 every 8 connection(s)
I20260806 04:43:36.777776 19314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:36.780560 19388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:36.784945 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 19314
I20260806 04:43:36.785109 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:36.785430 32410 external_mini_cluster.cc:1428] Setting key 6e6084fb85462e8cfd3a2ab56bc8d1f9
I20260806 04:43:36.786316 19388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678: Bootstrap starting.
I20260806 04:43:36.788717 19388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:36.789644 19388 log.cc:824] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:36.791612 19388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678: No bootstrap required, opened a new log
Aug 06 04:43:36 dist-test-slave-vkjp krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:43:36.795583 19388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b01d165cd5041a6a16bbaf6d9add678" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45733 } }
I20260806 04:43:36.795902 19388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:36.795966 19388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b01d165cd5041a6a16bbaf6d9add678, State: Initialized, Role: FOLLOWER
I20260806 04:43:36.796542 19388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b01d165cd5041a6a16bbaf6d9add678" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45733 } }
I20260806 04:43:36.796712 19388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:36.796772 19388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:36.796868 19388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:36.798026 19388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b01d165cd5041a6a16bbaf6d9add678" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45733 } }
I20260806 04:43:36.798447 19388 leader_election.cc:302] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b01d165cd5041a6a16bbaf6d9add678; no voters:
I20260806 04:43:36.798816 19388 leader_election.cc:288] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:36.798990 19393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:36.799328 19393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [term 1 LEADER]: Becoming Leader. State: Replica: 0b01d165cd5041a6a16bbaf6d9add678, State: Running, Role: LEADER
I20260806 04:43:36.799685 19393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b01d165cd5041a6a16bbaf6d9add678" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45733 } }
I20260806 04:43:36.800258 19388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:36.801003 19396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b01d165cd5041a6a16bbaf6d9add678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b01d165cd5041a6a16bbaf6d9add678" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45733 } } }
I20260806 04:43:36.801003 19393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b01d165cd5041a6a16bbaf6d9add678. Latest consensus state: current_term: 1 leader_uuid: "0b01d165cd5041a6a16bbaf6d9add678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b01d165cd5041a6a16bbaf6d9add678" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 45733 } } }
I20260806 04:43:36.801185 19393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:36.801185 19396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:36.801394 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:36.786827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.31.166.190:45733)
0806 04:43:36.787276 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:36.787291 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:36.787327 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:43:36.787957 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:36.787974 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:36.788290 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:36.788563 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:36.789509 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:36.790570 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:36.791526 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:36.792047 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:36.792233 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:36.795128 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:36.795134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:36.795152 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:36.795183 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:36.797336 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:36.797823 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:36.797830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:36.797836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:36.798003 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:36.798422 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:36.798425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:36.798427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:36.799021 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:36.799299 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:36.799758 (+ 459us) server_negotiation.cc:299] Negotiation successful
0806 04:43:36.800152 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":141,"threads_started":1}
I20260806 04:43:36.801636 19403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:36.804777 19403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:36.810367 19403 catalog_manager.cc:1347] Generated new cluster ID: f06660fd030d438aaf534a714f487c01
I20260806 04:43:36.810480 19403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:36.824018 19403 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:36.825291 19403 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:36.844076 19403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678: Generated new TSK 0
I20260806 04:43:36.844877 19403 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:43:36.881709 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:45733
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:36.996886 19415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:36.997241 19415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:36.997292 19415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:37.001684 19415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:37.001754 19415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:37.001816 19415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:43:37.007372 19415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:45733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:37.008428 19415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:37.009397 19415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:37.019163 19420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:37.019237 19421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:37.019292 19415 server_base.cc:1034] running on GCE node
W20260806 04:43:37.019163 19423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:37.019893 19415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:37.020473 19415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:37.021646 19415 hybrid_clock.cc:648] HybridClock initialized: now 1785991417021625 us; error 31 us; skew 500 ppm
Aug 06 04:43:37 dist-test-slave-vkjp krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991417, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:37.024595 19415 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:43:37.025514 19415 webserver.cc:460] Webserver started at http://127.31.166.129:42053/ using document root <none> and password file <none>
I20260806 04:43:37.026108 19415 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:37.026204 19415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:37.026453 19415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:37.028628 19415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "3ccda8e4c6ee4cada32e4467528d4758"
format_stamp: "Formatted at 2026-08-06 04:43:37 on dist-test-slave-vkjp"
server_key: "899a158ede4a8c314694b4d07b18a49a"
server_key_iv: "3eb78911891d2d28b49538e048c1259f"
server_key_version: "encryptionkey@0"
I20260806 04:43:37.029212 19415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "3ccda8e4c6ee4cada32e4467528d4758"
format_stamp: "Formatted at 2026-08-06 04:43:37 on dist-test-slave-vkjp"
server_key: "899a158ede4a8c314694b4d07b18a49a"
server_key_iv: "3eb78911891d2d28b49538e048c1259f"
server_key_version: "encryptionkey@0"
I20260806 04:43:37.033154 19415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:43:37.035657 19430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:37.037014 19415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:43:37.037184 19415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "3ccda8e4c6ee4cada32e4467528d4758"
format_stamp: "Formatted at 2026-08-06 04:43:37 on dist-test-slave-vkjp"
server_key: "899a158ede4a8c314694b4d07b18a49a"
server_key_iv: "3eb78911891d2d28b49538e048c1259f"
server_key_version: "encryptionkey@0"
I20260806 04:43:37.037313 19415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:37.052810 19415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:37.056859 19415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:37.057065 19415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:37.057730 19415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:37.058602 19415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:37.058656 19415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:37.058701 19415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:37.058717 19415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:37.068729 19415 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:45259
I20260806 04:43:37.068754 19555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:45259 every 8 connection(s)
I20260806 04:43:37.069684 19415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:43:37.078460 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 19415
I20260806 04:43:37.078621 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:43:37.078962 32410 external_mini_cluster.cc:1428] Setting key a3b03fa4f460a61b6cbe9efa51328eb0
Aug 06 04:43:37 dist-test-slave-vkjp krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991417, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:43:37.084796 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.071617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:41063 (local address 127.31.166.190:45733)
0806 04:43:37.071788 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:37.071792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:37.072734 (+ 942us) server_negotiation.cc:407] Connection header received
0806 04:43:37.074139 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:37.074160 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:37.074235 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:37.074454 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:37.075664 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.076194 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.077351 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.077597 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.077707 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.080591 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:37.080595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:37.080598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:37.080628 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:37.082375 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:37.082939 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:37.082944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:37.082947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:37.083047 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:37.083505 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:37.083511 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:37.083516 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:37.083768 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:37.083876 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:37.084383 (+ 507us) server_negotiation.cc:299] Negotiation successful
0806 04:43:37.084538 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 04:43:37.085498 19558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.072002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:45733 (local address 127.31.166.129:41063)
0806 04:43:37.072545 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:37.072606 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:37.073903 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:37.074651 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:37.074657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:37.075033 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:37.075512 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.075526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.076326 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.076330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.077216 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.077226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.077761 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.077765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.077932 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.078527 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:43:37.078551 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:37.080432 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:37.082527 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:37.082540 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:37.082552 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:37.082832 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:37.083217 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:37.083220 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:37.083222 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:37.083397 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:37.083879 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:37.083884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:37.084124 (+ 240us) client_negotiation.cc:769] Sending connection context
0806 04:43:37.084295 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 04:43:37.084559 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":110,"threads_started":1}
I20260806 04:43:37.086902 19556 heartbeater.cc:347] Connected to a master server at 127.31.166.190:45733
I20260806 04:43:37.087195 19556 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:37.087795 19556 heartbeater.cc:511] Master 127.31.166.190:45733 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
I20260806 04:43:37.089494 19347 ts_manager.cc:194] Registered new tserver with Master: 3ccda8e4c6ee4cada32e4467528d4758 (127.31.166.129:45259)
I20260806 04:43:37.090767 19347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:41063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:43:37.107646 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:45733
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:37.220221 19561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:37.220499 19561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:37.220571 19561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:37.224593 19561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:37.224679 19561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:37.224772 19561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:43:37.230019 19561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:45733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:37.231186 19561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:37.232254 19561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:37.242161 19567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:37.242157 19569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:37.242131 19566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:37.242573 19561 server_base.cc:1034] running on GCE node
I20260806 04:43:37.242980 19561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:37.243695 19561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:37.244875 19561 hybrid_clock.cc:648] HybridClock initialized: now 1785991417244856 us; error 32 us; skew 500 ppm
Aug 06 04:43:37 dist-test-slave-vkjp krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991417, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:37.247736 19561 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:43:37.248629 19561 webserver.cc:460] Webserver started at http://127.31.166.130:40575/ using document root <none> and password file <none>
I20260806 04:43:37.249164 19561 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:37.249223 19561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:37.249446 19561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:37.251531 19561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d"
format_stamp: "Formatted at 2026-08-06 04:43:37 on dist-test-slave-vkjp"
server_key: "382a854bd888137b9f0ecfbae7db3238"
server_key_iv: "09be55f30b2d8c75623464af434f8015"
server_key_version: "encryptionkey@0"
I20260806 04:43:37.252040 19561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d"
format_stamp: "Formatted at 2026-08-06 04:43:37 on dist-test-slave-vkjp"
server_key: "382a854bd888137b9f0ecfbae7db3238"
server_key_iv: "09be55f30b2d8c75623464af434f8015"
server_key_version: "encryptionkey@0"
I20260806 04:43:37.255923 19561 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:43:37.258265 19577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:37.259508 19561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:43:37.259599 19561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d"
format_stamp: "Formatted at 2026-08-06 04:43:37 on dist-test-slave-vkjp"
server_key: "382a854bd888137b9f0ecfbae7db3238"
server_key_iv: "09be55f30b2d8c75623464af434f8015"
server_key_version: "encryptionkey@0"
I20260806 04:43:37.259682 19561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:37.274000 19561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:37.277326 19561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:37.277486 19561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:37.278014 19561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:37.278777 19561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:37.278823 19561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:37.278872 19561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:37.278888 19561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:37.287676 19561 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:37253
I20260806 04:43:37.287703 19696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:37253 every 8 connection(s)
I20260806 04:43:37.288566 19561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:43:37.294165 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 19561
I20260806 04:43:37.294322 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:43:37.294610 32410 external_mini_cluster.cc:1428] Setting key 1200af61f2a23951b524e590cdf11812
Aug 06 04:43:37 dist-test-slave-vkjp krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991417, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:43:37.302977 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.290429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:53279 (local address 127.31.166.190:45733)
0806 04:43:37.290612 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:37.290617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:37.291429 (+ 812us) server_negotiation.cc:407] Connection header received
0806 04:43:37.292692 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:37.292695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:37.292742 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:37.292816 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:37.293957 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.294601 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.295914 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.296163 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.296278 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.298710 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:37.298714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:37.298716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:37.298750 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:37.300518 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:37.301049 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:37.301053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:37.301055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:37.301144 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:37.301643 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:37.301647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:37.301649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:37.301857 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:37.301967 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:37.302671 (+ 704us) server_negotiation.cc:299] Negotiation successful
0806 04:43:37.302799 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 04:43:37.303817 19699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.290760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:45733 (local address 127.31.166.130:53279)
0806 04:43:37.291270 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:37.291313 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:37.292506 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:37.292986 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:37.292992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:37.293323 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:37.293786 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.293799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.294731 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.294734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.295767 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.295777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.296320 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.296324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.296499 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.297108 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:43:37.297124 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:37.298544 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:37.300621 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:37.300628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:37.300642 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:37.300913 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:37.301270 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:37.301274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:37.301276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:37.301493 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:37.301966 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:37.301974 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:37.302347 (+ 373us) client_negotiation.cc:769] Sending connection context
0806 04:43:37.302579 (+ 232us) client_negotiation.cc:240] Negotiation successful
0806 04:43:37.302926 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:43:37.304993 19697 heartbeater.cc:347] Connected to a master server at 127.31.166.190:45733
I20260806 04:43:37.305259 19697 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:37.305823 19697 heartbeater.cc:511] Master 127.31.166.190:45733 requested a full tablet report, sending...
I20260806 04:43:37.306945 19347 ts_manager.cc:194] Registered new tserver with Master: 35b6c0de9bf5425ba1dd0aa6386bda2d (127.31.166.130:37253)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
I20260806 04:43:37.307699 19347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:53279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:43:37.324807 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:45733
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:42733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:37.435725 19703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:37.435978 19703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:37.436021 19703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:37.440414 19703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:37.440482 19703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:37.440543 19703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:43:37.446066 19703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:42733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:45733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:37.447125 19703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:37.448115 19703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:37.457942 19708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:37.457914 19709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:37.457924 19711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:37.458323 19703 server_base.cc:1034] running on GCE node
I20260806 04:43:37.458734 19703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:37.459404 19703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:37.460606 19703 hybrid_clock.cc:648] HybridClock initialized: now 1785991417460577 us; error 44 us; skew 500 ppm
Aug 06 04:43:37 dist-test-slave-vkjp krb5kdc[19299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991417, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:37.469600 19703 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:43:37.470468 19703 webserver.cc:460] Webserver started at http://127.31.166.131:34659/ using document root <none> and password file <none>
I20260806 04:43:37.471030 19703 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:37.471153 19703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:37.471395 19703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:37.473390 19703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e"
format_stamp: "Formatted at 2026-08-06 04:43:37 on dist-test-slave-vkjp"
server_key: "60be260a8e22dc609c30702ad1a21f1b"
server_key_iv: "a8d754b7fc44c198c86bbafc3c1e2dc4"
server_key_version: "encryptionkey@0"
I20260806 04:43:37.473897 19703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e"
format_stamp: "Formatted at 2026-08-06 04:43:37 on dist-test-slave-vkjp"
server_key: "60be260a8e22dc609c30702ad1a21f1b"
server_key_iv: "a8d754b7fc44c198c86bbafc3c1e2dc4"
server_key_version: "encryptionkey@0"
I20260806 04:43:37.477643 19703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:43:37.480070 19718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:37.481334 19703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:43:37.481457 19703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e"
format_stamp: "Formatted at 2026-08-06 04:43:37 on dist-test-slave-vkjp"
server_key: "60be260a8e22dc609c30702ad1a21f1b"
server_key_iv: "a8d754b7fc44c198c86bbafc3c1e2dc4"
server_key_version: "encryptionkey@0"
I20260806 04:43:37.481562 19703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:37.494261 19703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:37.497622 19703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:37.497839 19703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:37.498467 19703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:37.499380 19703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:37.499428 19703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:37.499495 19703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:37.499536 19703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:37.508827 19703 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:34739
I20260806 04:43:37.508858 19840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:34739 every 8 connection(s)
I20260806 04:43:37.509714 19703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:43:37.511456 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 19703
I20260806 04:43:37.511607 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:43:37.511926 32410 external_mini_cluster.cc:1428] Setting key 4a940c20a408f64ab61a5a00fb883531
Aug 06 04:43:37 dist-test-slave-vkjp krb5kdc[19299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991417, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:43:37.523921 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.511634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:37963 (local address 127.31.166.190:45733)
0806 04:43:37.511780 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:37.511784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:37.512690 (+ 906us) server_negotiation.cc:407] Connection header received
0806 04:43:37.514453 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:37.514456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:37.514518 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:37.514606 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:37.515824 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.516347 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.517456 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.517676 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.517757 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.520282 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:37.520285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:37.520287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:37.520312 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:37.521799 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:37.522386 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:37.522389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:37.522390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:37.522466 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:37.522799 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:37.522801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:37.522802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:37.522994 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:37.523127 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:37.523672 (+ 545us) server_negotiation.cc:299] Negotiation successful
0806 04:43:37.523780 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:43:37.524837 19843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.512032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:45733 (local address 127.31.166.131:37963)
0806 04:43:37.512528 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:37.512571 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:37.514203 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:37.514751 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:37.514757 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:37.515209 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:37.515684 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.515696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.516464 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.516467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.517342 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.517352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.517806 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.517809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.517962 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.518523 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:43:37.518539 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:37.520123 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:37.521922 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:37.521928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:37.521940 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:37.522248 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:37.522552 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:37.522554 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:37.522556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:37.522710 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:37.523163 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:37.523168 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:37.523453 (+ 285us) client_negotiation.cc:769] Sending connection context
0806 04:43:37.523624 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 04:43:37.523900 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260806 04:43:37.526119 19841 heartbeater.cc:347] Connected to a master server at 127.31.166.190:45733
I20260806 04:43:37.526405 19841 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:37.526916 19841 heartbeater.cc:511] Master 127.31.166.190:45733 requested a full tablet report, sending...
I20260806 04:43:37.527967 19347 ts_manager.cc:194] Registered new tserver with Master: 738ba1fa4f5d4af283bbd32b2eb1f81e (127.31.166.131:34739)
I20260806 04:43:37.528566 19347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:37963
I20260806 04:43:37.538086 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:43:37.548115 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.540235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.31.166.190:45733)
0806 04:43:37.540407 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:37.540411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:37.540536 (+ 125us) server_negotiation.cc:407] Connection header received
0806 04:43:37.540666 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:37.540669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:37.540734 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:37.540830 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:37.541610 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.542190 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.543015 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.543291 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.543403 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.544972 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:37.544976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:37.544978 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:37.545012 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:37.546242 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:37.546757 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:37.546763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:37.546764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:37.546843 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:37.547209 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:37.547212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:37.547214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:37.547423 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:37.547581 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:37.547867 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 04:43:37.547964 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 04:43:37.551136 19347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:43:37.554152 19347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:43:37.568959 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.562682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:34739 (local address 127.0.0.1:46200)
0806 04:43:37.563688 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:37.563703 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:37.563787 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:37.564642 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:37.564646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:37.564668 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:37.564862 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.564870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.566083 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.566088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.567333 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.567343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.568392 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.568396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.568501 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.568541 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 04:43:37.568658 (+ 117us) client_negotiation.cc:240] Negotiation successful
0806 04:43:37.568776 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":23,"thread_start_us":79,"threads_started":1}
I20260806 04:43:37.568971 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.563590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46200 (local address 127.31.166.131:34739)
0806 04:43:37.564073 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:37.564087 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:37.564114 (+ 27us) server_negotiation.cc:407] Connection header received
0806 04:43:37.564209 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:37.564214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:37.564462 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:37.564634 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:37.565001 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.565950 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.567460 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.568255 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.568398 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.568532 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:37.568706 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 04:43:37.568814 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":131,"threads_started":1}
I20260806 04:43:37.569584 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.563003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:45259 (local address 127.0.0.1:51520)
0806 04:43:37.564916 (+ 1913us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:37.564932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:37.565033 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:37.565471 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:37.565474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:37.565493 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:37.565576 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.565582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.566907 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.566910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.567978 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.567985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.569176 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.569182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.569287 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.569309 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:43:37.569387 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 04:43:37.569453 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1843,"thread_start_us":107,"threads_started":1}
I20260806 04:43:37.570012 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.563855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.31.166.129:45259)
0806 04:43:37.564205 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:37.564219 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:37.565015 (+ 796us) server_negotiation.cc:407] Connection header received
0806 04:43:37.565142 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:37.565147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:37.565320 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:37.565470 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:37.565671 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.566743 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.568092 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.568861 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.568976 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.569111 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:37.569759 (+ 648us) server_negotiation.cc:299] Negotiation successful
0806 04:43:37.569859 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":93,"threads_started":1}
I20260806 04:43:37.570132 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.562664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:37253 (local address 127.0.0.1:39762)
0806 04:43:37.563398 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:37.563450 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:37.563594 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:37.564771 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:37.564774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:37.564791 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:37.564919 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.564927 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.566420 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.566423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.567657 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.567667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.568788 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.568792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.568897 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.568920 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:43:37.568972 (+ 52us) client_negotiation.cc:240] Negotiation successful
0806 04:43:37.569054 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":23,"thread_start_us":99,"threads_started":1}
I20260806 04:43:37.570839 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.562824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.31.166.130:37253)
0806 04:43:37.564370 (+ 1546us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:37.564381 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:37.564402 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:43:37.564472 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:37.564479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:37.564644 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:37.564776 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:37.565051 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.566255 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.567781 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.568644 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.570407 (+ 1763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.570524 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:37.570570 (+ 46us) server_negotiation.cc:299] Negotiation successful
0806 04:43:37.570651 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1438,"thread_start_us":141,"threads_started":1}
I20260806 04:43:37.572239 19770 tablet_service.cc:1467] Processing CreateTablet for tablet f8e6300a824842afb06376ae76835f69 (DEFAULT_TABLE table=test-table [id=5fbd62847a7b4c7683f65e77319b0afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:37.572994 19629 tablet_service.cc:1467] Processing CreateTablet for tablet f8e6300a824842afb06376ae76835f69 (DEFAULT_TABLE table=test-table [id=5fbd62847a7b4c7683f65e77319b0afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:37.573412 19770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8e6300a824842afb06376ae76835f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:37.572992 19483 tablet_service.cc:1467] Processing CreateTablet for tablet f8e6300a824842afb06376ae76835f69 (DEFAULT_TABLE table=test-table [id=5fbd62847a7b4c7683f65e77319b0afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:37.573941 19629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8e6300a824842afb06376ae76835f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:37.573972 19483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8e6300a824842afb06376ae76835f69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:37.580024 19858 tablet_bootstrap.cc:492] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d: Bootstrap starting.
I20260806 04:43:37.581405 19859 tablet_bootstrap.cc:492] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758: Bootstrap starting.
I20260806 04:43:37.582207 19858 tablet_bootstrap.cc:654] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:37.583199 19859 tablet_bootstrap.cc:654] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:37.583446 19858 log.cc:824] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:37.583469 19860 tablet_bootstrap.cc:492] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e: Bootstrap starting.
I20260806 04:43:37.584507 19859 log.cc:824] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:37.585691 19860 tablet_bootstrap.cc:654] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:37.586169 19858 tablet_bootstrap.cc:492] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d: No bootstrap required, opened a new log
I20260806 04:43:37.586365 19858 ts_tablet_manager.cc:1397] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 04:43:37.586824 19859 tablet_bootstrap.cc:492] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758: No bootstrap required, opened a new log
I20260806 04:43:37.587024 19859 ts_tablet_manager.cc:1397] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260806 04:43:37.591696 19860 log.cc:824] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:37.591636 19859 raft_consensus.cc:348] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37253 } } peers { permanent_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34739 } } peers { permanent_uuid: "3ccda8e4c6ee4cada32e4467528d4758" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45259 } }
I20260806 04:43:37.591936 19859 raft_consensus.cc:374] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:37.591996 19859 raft_consensus.cc:729] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccda8e4c6ee4cada32e4467528d4758, State: Initialized, Role: FOLLOWER
I20260806 04:43:37.591817 19858 raft_consensus.cc:348] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37253 } } peers { permanent_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34739 } } peers { permanent_uuid: "3ccda8e4c6ee4cada32e4467528d4758" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45259 } }
I20260806 04:43:37.592082 19858 raft_consensus.cc:374] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:37.592144 19858 raft_consensus.cc:729] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b6c0de9bf5425ba1dd0aa6386bda2d, State: Initialized, Role: FOLLOWER
I20260806 04:43:37.592599 19858 consensus_queue.cc:260] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37253 } } peers { permanent_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34739 } } peers { permanent_uuid: "3ccda8e4c6ee4cada32e4467528d4758" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45259 } }
I20260806 04:43:37.592697 19859 consensus_queue.cc:260] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37253 } } peers { permanent_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34739 } } peers { permanent_uuid: "3ccda8e4c6ee4cada32e4467528d4758" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45259 } }
I20260806 04:43:37.593636 19858 ts_tablet_manager.cc:1428] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 04:43:37.593641 19859 ts_tablet_manager.cc:1428] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 04:43:37.593854 19860 tablet_bootstrap.cc:492] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e: No bootstrap required, opened a new log
I20260806 04:43:37.594017 19697 heartbeater.cc:503] Master 127.31.166.190:45733 was elected leader, sending a full tablet report...
I20260806 04:43:37.594043 19860 ts_tablet_manager.cc:1397] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260806 04:43:37.594137 19556 heartbeater.cc:503] Master 127.31.166.190:45733 was elected leader, sending a full tablet report...
I20260806 04:43:37.598212 19860 raft_consensus.cc:348] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37253 } } peers { permanent_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34739 } } peers { permanent_uuid: "3ccda8e4c6ee4cada32e4467528d4758" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45259 } }
I20260806 04:43:37.598531 19860 raft_consensus.cc:374] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:37.598609 19860 raft_consensus.cc:729] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 738ba1fa4f5d4af283bbd32b2eb1f81e, State: Initialized, Role: FOLLOWER
I20260806 04:43:37.599133 19860 consensus_queue.cc:260] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37253 } } peers { permanent_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34739 } } peers { permanent_uuid: "3ccda8e4c6ee4cada32e4467528d4758" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45259 } }
I20260806 04:43:37.599824 19841 heartbeater.cc:503] Master 127.31.166.190:45733 was elected leader, sending a full tablet report...
I20260806 04:43:37.600234 19860 ts_tablet_manager.cc:1428] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260806 04:43:37.760732 19842 tablet.cc:2369] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:43:37.789710 19698 tablet.cc:2369] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:43:37.821174 19557 tablet.cc:2369] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:43:37.937191 19866 raft_consensus.cc:482] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:43:37.937533 19866 raft_consensus.cc:504] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37253 } } peers { permanent_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34739 } } peers { permanent_uuid: "3ccda8e4c6ee4cada32e4467528d4758" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45259 } }
I20260806 04:43:37.939304 19866 leader_election.cc:288] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35b6c0de9bf5425ba1dd0aa6386bda2d (127.31.166.130:37253), 3ccda8e4c6ee4cada32e4467528d4758 (127.31.166.129:45259)
I20260806 04:43:37.944145 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.940220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:40129 (local address 127.31.166.129:45259)
0806 04:43:37.940395 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:37.940399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:37.940412 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:43:37.940463 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:37.940465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:37.940509 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:37.940603 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:37.940958 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.941675 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.943119 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.943530 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.943626 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.943708 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:37.943924 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 04:43:37.944007 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 04:43:37.944145 19843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.939510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:37253 (local address 127.31.166.131:44215)
0806 04:43:37.939830 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:37.939849 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:37.939964 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:37.941102 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:37.941105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:37.941130 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:37.941242 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.941248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.942158 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.942161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.942977 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.942987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.943716 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.943721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.943817 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.943834 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 04:43:37.943937 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 04:43:37.944007 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":1}
I20260806 04:43:37.944165 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.940225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:44215 (local address 127.31.166.130:37253)
0806 04:43:37.940841 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:37.940849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:37.940867 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:43:37.940927 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:37.940931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:37.940979 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:37.941059 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:37.941383 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.942050 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.943149 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.943566 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.943714 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.943786 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:37.943968 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 04:43:37.944045 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507}
I20260806 04:43:37.944186 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:37.939690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:45259 (local address 127.31.166.131:40129)
0806 04:43:37.940097 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:37.940114 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:37.940205 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:37.940619 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:37.940633 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:37.940652 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:37.940807 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.940814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.941786 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.941790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.942963 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:37.942973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:37.943656 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:37.943664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:37.943771 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:37.943801 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 04:43:37.943885 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 04:43:37.943986 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":21,"thread_start_us":126,"threads_started":1}
I20260806 04:43:37.944875 19649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8e6300a824842afb06376ae76835f69" candidate_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d" is_pre_election: true
I20260806 04:43:37.945309 19649 raft_consensus.cc:2454] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 738ba1fa4f5d4af283bbd32b2eb1f81e in term 0.
I20260806 04:43:37.945376 19504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8e6300a824842afb06376ae76835f69" candidate_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccda8e4c6ee4cada32e4467528d4758" is_pre_election: true
I20260806 04:43:37.945711 19504 raft_consensus.cc:2454] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 738ba1fa4f5d4af283bbd32b2eb1f81e in term 0.
I20260806 04:43:37.945930 19719 leader_election.cc:302] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b6c0de9bf5425ba1dd0aa6386bda2d, 738ba1fa4f5d4af283bbd32b2eb1f81e; no voters:
I20260806 04:43:37.946242 19866 raft_consensus.cc:2781] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:43:37.946338 19866 raft_consensus.cc:482] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:43:37.946369 19866 raft_consensus.cc:3037] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:37.947562 19866 raft_consensus.cc:504] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37253 } } peers { permanent_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34739 } } peers { permanent_uuid: "3ccda8e4c6ee4cada32e4467528d4758" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45259 } }
I20260806 04:43:37.947949 19866 leader_election.cc:288] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [CANDIDATE]: Term 1 election: Requested vote from peers 35b6c0de9bf5425ba1dd0aa6386bda2d (127.31.166.130:37253), 3ccda8e4c6ee4cada32e4467528d4758 (127.31.166.129:45259)
I20260806 04:43:37.948318 19649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8e6300a824842afb06376ae76835f69" candidate_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d"
I20260806 04:43:37.948359 19504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8e6300a824842afb06376ae76835f69" candidate_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ccda8e4c6ee4cada32e4467528d4758"
I20260806 04:43:37.948438 19649 raft_consensus.cc:3037] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:37.948503 19504 raft_consensus.cc:3037] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:37.949553 19649 raft_consensus.cc:2454] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 738ba1fa4f5d4af283bbd32b2eb1f81e in term 1.
I20260806 04:43:37.949681 19504 raft_consensus.cc:2454] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 738ba1fa4f5d4af283bbd32b2eb1f81e in term 1.
I20260806 04:43:37.949967 19719 leader_election.cc:302] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35b6c0de9bf5425ba1dd0aa6386bda2d, 738ba1fa4f5d4af283bbd32b2eb1f81e; no voters:
I20260806 04:43:37.950145 19866 raft_consensus.cc:2781] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:37.950403 19866 raft_consensus.cc:686] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [term 1 LEADER]: Becoming Leader. State: Replica: 738ba1fa4f5d4af283bbd32b2eb1f81e, State: Running, Role: LEADER
I20260806 04:43:37.950731 19866 consensus_queue.cc:237] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37253 } } peers { permanent_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34739 } } peers { permanent_uuid: "3ccda8e4c6ee4cada32e4467528d4758" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45259 } }
I20260806 04:43:37.953902 19347 catalog_manager.cc:5495] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e reported cstate change: term changed from 0 to 1, leader changed from <none> to 738ba1fa4f5d4af283bbd32b2eb1f81e (127.31.166.131). New cstate: current_term: 1 leader_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "738ba1fa4f5d4af283bbd32b2eb1f81e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ccda8e4c6ee4cada32e4467528d4758" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45259 } health_report { overall_health: UNKNOWN } } }
I20260806 04:43:38.026666 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:38.022691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46216 (local address 127.31.166.131:34739)
0806 04:43:38.022878 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:38.022886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:38.022904 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:43:38.022986 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:38.022989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:38.023050 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:38.023209 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:43:38.023547 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.024069 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.024992 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.025221 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.025347 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:38.025694 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:43:38.026129 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:43:38.026306 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:38.026451 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 04:43:38.026498 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 04:43:38.032794 19649 raft_consensus.cc:1261] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d [term 1 FOLLOWER]: Refusing update from remote peer 738ba1fa4f5d4af283bbd32b2eb1f81e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:43:38.032794 19504 raft_consensus.cc:1261] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758 [term 1 FOLLOWER]: Refusing update from remote peer 738ba1fa4f5d4af283bbd32b2eb1f81e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:43:38.033497 19866 consensus_queue.cc:1035] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b6c0de9bf5425ba1dd0aa6386bda2d" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:43:38.033747 19868 consensus_queue.cc:1035] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ccda8e4c6ee4cada32e4467528d4758" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:43:38.043732 19872 mvcc.cc:205] Tried to move back new op lower bound from 7315420848253935616 to 7315420847928958976. Current Snapshot: MvccSnapshot[applied={T|T < 7315420848253935616}]
I20260806 04:43:38.050912 19346 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:43:38.051143 19346 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:43:38.053767 19346 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0b01d165cd5041a6a16bbaf6d9add678: Sending DeleteTablet for 3 replicas of tablet f8e6300a824842afb06376ae76835f69
I20260806 04:43:38.054610 19770 tablet_service.cc:1514] Processing DeleteTablet for tablet f8e6300a824842afb06376ae76835f69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:38 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:46200
I20260806 04:43:38.054615 19629 tablet_service.cc:1514] Processing DeleteTablet for tablet f8e6300a824842afb06376ae76835f69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:38 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:39762
I20260806 04:43:38.054651 19483 tablet_service.cc:1514] Processing DeleteTablet for tablet f8e6300a824842afb06376ae76835f69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:38 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:51520
I20260806 04:43:38.055156 19881 tablet_replica.cc:330] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d: stopping tablet replica
I20260806 04:43:38.055538 19881 raft_consensus.cc:2229] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:43:38.055657 19884 tablet_replica.cc:330] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758: stopping tablet replica
I20260806 04:43:38.055864 19881 raft_consensus.cc:2258] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:43:38.055968 19884 raft_consensus.cc:2229] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:43:38.056211 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 19415
I20260806 04:43:38.056386 19884 raft_consensus.cc:2258] T f8e6300a824842afb06376ae76835f69 P 3ccda8e4c6ee4cada32e4467528d4758 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:43:38.057447 19881 ts_tablet_manager.cc:1905] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:43:38.059320 19883 tablet_replica.cc:330] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e: stopping tablet replica
I20260806 04:43:38.059604 19883 raft_consensus.cc:2229] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:43:38.060179 19883 raft_consensus.cc:2258] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:43:38.060312 19881 ts_tablet_manager.cc:1918] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:43:38.060424 19881 log.cc:1196] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/f8e6300a824842afb06376ae76835f69
I20260806 04:43:38.060738 19881 ts_tablet_manager.cc:1939] T f8e6300a824842afb06376ae76835f69 P 35b6c0de9bf5425ba1dd0aa6386bda2d: Deleting consensus metadata
I20260806 04:43:38.061957 19331 catalog_manager.cc:4841] TS 35b6c0de9bf5425ba1dd0aa6386bda2d (127.31.166.130:37253): tablet f8e6300a824842afb06376ae76835f69 (table test-table [id=5fbd62847a7b4c7683f65e77319b0afd]) successfully deleted
I20260806 04:43:38.064589 19883 ts_tablet_manager.cc:1905] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:43:38.068308 19883 ts_tablet_manager.cc:1918] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260806 04:43:38.068382 19333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:45259 (error 108)
I20260806 04:43:38.068539 19883 log.cc:1196] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/f8e6300a824842afb06376ae76835f69
I20260806 04:43:38.069020 19883 ts_tablet_manager.cc:1939] T f8e6300a824842afb06376ae76835f69 P 738ba1fa4f5d4af283bbd32b2eb1f81e: Deleting consensus metadata
I20260806 04:43:38.069306 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 19561
I20260806 04:43:38.075577 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:38.071956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:45259 (local address 127.0.0.1:51534)
0806 04:43:38.075375 (+ 3419us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:38.075461 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3300}
I20260806 04:43:38.075989 19334 catalog_manager.cc:4841] TS 738ba1fa4f5d4af283bbd32b2eb1f81e (127.31.166.131:34739): tablet f8e6300a824842afb06376ae76835f69 (table test-table [id=5fbd62847a7b4c7683f65e77319b0afd]) successfully deleted
W20260806 04:43:38.076236 19333 catalog_manager.cc:4584] TS 3ccda8e4c6ee4cada32e4467528d4758 (127.31.166.129:45259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f8e6300a824842afb06376ae76835f69: Network error: Client connection negotiation failed: client connection to 127.31.166.129:45259: connect: Connection refused (error 111)
I20260806 04:43:38.080590 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 19703
I20260806 04:43:38.086664 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 19314
2026-08-06T04:43:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2765 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T04:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:38Z Disabled control of system clock
I20260806 04:43:38.119369 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:33331
--webserver_interface=127.31.166.190
--webserver_port=0
--builtin_ntp_servers=127.31.166.148:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:33331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:43:38.232676 19890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:38.232971 19890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:38.233008 19890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:38.237134 19890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:38.237195 19890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:38.237211 19890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:38.237229 19890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:38.237244 19890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:38.242014 19890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:33331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:33331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:38.243058 19890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:38.244499 19890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:38.252992 19898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:38.252986 19896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:38.252992 19895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:38.253235 19890 server_base.cc:1034] running on GCE node
I20260806 04:43:38.253875 19890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:38.254976 19890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:38.256155 19890 hybrid_clock.cc:648] HybridClock initialized: now 1785991418256131 us; error 36 us; skew 500 ppm
I20260806 04:43:38.258102 19890 webserver.cc:460] Webserver started at http://127.31.166.190:35639/ using document root <none> and password file <none>
I20260806 04:43:38.258687 19890 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:38.258747 19890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:38.258965 19890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:38.261037 19890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "8dccb1ba8e654ce9833e7630d5bd3cbb"
format_stamp: "Formatted at 2026-08-06 04:43:38 on dist-test-slave-vkjp"
server_key: "74c30a74311fe8ee29e4ac0c09b2d246"
server_key_iv: "d53ba7a06831f45d144700759f2dafa8"
server_key_version: "encryptionkey@0"
I20260806 04:43:38.261615 19890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "8dccb1ba8e654ce9833e7630d5bd3cbb"
format_stamp: "Formatted at 2026-08-06 04:43:38 on dist-test-slave-vkjp"
server_key: "74c30a74311fe8ee29e4ac0c09b2d246"
server_key_iv: "d53ba7a06831f45d144700759f2dafa8"
server_key_version: "encryptionkey@0"
I20260806 04:43:38.265470 19890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:43:38.267973 19905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:38.269371 19890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:43:38.269526 19890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "8dccb1ba8e654ce9833e7630d5bd3cbb"
format_stamp: "Formatted at 2026-08-06 04:43:38 on dist-test-slave-vkjp"
server_key: "74c30a74311fe8ee29e4ac0c09b2d246"
server_key_iv: "d53ba7a06831f45d144700759f2dafa8"
server_key_version: "encryptionkey@0"
I20260806 04:43:38.269654 19890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:38.284696 19890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:38.285467 19890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:38.285676 19890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:38.293332 19890 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:33331
I20260806 04:43:38.293355 19959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:33331 every 8 connection(s)
I20260806 04:43:38.294338 19890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:38.295387 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 19890
I20260806 04:43:38.295538 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:38.295888 32410 external_mini_cluster.cc:1428] Setting key 5ee9205e1b35c2c403ce86262398f86c
I20260806 04:43:38.298233 19960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:38.304224 19960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb: Bootstrap starting.
I20260806 04:43:38.305797 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:38.297444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53622 (local address 127.31.166.190:33331)
0806 04:43:38.298115 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:38.298130 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:38.298165 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:43:38.298856 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:38.298873 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:38.299319 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:38.299664 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:38.300096 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.301083 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.302010 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.302836 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.303008 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:38.303992 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:38.303997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:38.304010 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:38.304479 (+ 469us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:38.304538 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:38.304559 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:38.304668 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:38.304826 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 04:43:38.305097 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":150,"threads_started":1}
I20260806 04:43:38.306593 19960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:38.307538 19960 log.cc:824] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:38.309777 19960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb: No bootstrap required, opened a new log
I20260806 04:43:38.313539 19960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dccb1ba8e654ce9833e7630d5bd3cbb" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33331 } }
I20260806 04:43:38.313771 19960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:38.313807 19960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dccb1ba8e654ce9833e7630d5bd3cbb, State: Initialized, Role: FOLLOWER
I20260806 04:43:38.314169 19960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dccb1ba8e654ce9833e7630d5bd3cbb" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33331 } }
I20260806 04:43:38.314278 19960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:38.314332 19960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:38.314401 19960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:38.315627 19960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dccb1ba8e654ce9833e7630d5bd3cbb" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33331 } }
I20260806 04:43:38.315934 19960 leader_election.cc:302] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dccb1ba8e654ce9833e7630d5bd3cbb; no voters:
I20260806 04:43:38.316203 19960 leader_election.cc:288] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:38.316342 19965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:38.316614 19965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [term 1 LEADER]: Becoming Leader. State: Replica: 8dccb1ba8e654ce9833e7630d5bd3cbb, State: Running, Role: LEADER
I20260806 04:43:38.316915 19965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dccb1ba8e654ce9833e7630d5bd3cbb" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33331 } }
I20260806 04:43:38.317584 19960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:38.318634 19966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dccb1ba8e654ce9833e7630d5bd3cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dccb1ba8e654ce9833e7630d5bd3cbb" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33331 } } }
I20260806 04:43:38.318781 19966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:38.319154 19974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:38.319794 19967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dccb1ba8e654ce9833e7630d5bd3cbb. Latest consensus state: current_term: 1 leader_uuid: "8dccb1ba8e654ce9833e7630d5bd3cbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dccb1ba8e654ce9833e7630d5bd3cbb" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 33331 } } }
I20260806 04:43:38.319916 19967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:38.322386 19974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:38.328783 19974 catalog_manager.cc:1347] Generated new cluster ID: 7c22448d4c8c49a8be41ac85e053d329
I20260806 04:43:38.328872 19974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:38.338038 19974 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:38.339461 19974 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:38.348613 19974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb: Generated new TSK 0
I20260806 04:43:38.349459 19974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:43:38.357775 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33331
--builtin_ntp_servers=127.31.166.148:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:43:38.471889 19984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:38.472187 19984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:38.472227 19984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:38.476438 19984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:38.476496 19984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:38.476552 19984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:43:38.481310 19984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:38.482394 19984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:38.483801 19984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:38.493708 19989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:38.493692 19990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:38.493661 19992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:38.494096 19984 server_base.cc:1034] running on GCE node
I20260806 04:43:38.494555 19984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:38.495280 19984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:38.496497 19984 hybrid_clock.cc:648] HybridClock initialized: now 1785991418496487 us; error 32 us; skew 500 ppm
I20260806 04:43:38.498492 19984 webserver.cc:460] Webserver started at http://127.31.166.129:43293/ using document root <none> and password file <none>
I20260806 04:43:38.499153 19984 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:38.499249 19984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:38.499490 19984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:38.501629 19984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "df63c372263a4d27877216eb9201f008"
format_stamp: "Formatted at 2026-08-06 04:43:38 on dist-test-slave-vkjp"
server_key: "b5652ef88f67bb64de789338abddfc28"
server_key_iv: "10d1b50b2dd6938eb3068b7fd5e6a985"
server_key_version: "encryptionkey@0"
I20260806 04:43:38.502211 19984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "df63c372263a4d27877216eb9201f008"
format_stamp: "Formatted at 2026-08-06 04:43:38 on dist-test-slave-vkjp"
server_key: "b5652ef88f67bb64de789338abddfc28"
server_key_iv: "10d1b50b2dd6938eb3068b7fd5e6a985"
server_key_version: "encryptionkey@0"
I20260806 04:43:38.506043 19984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:43:38.508531 19998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:38.509748 19984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:43:38.509898 19984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "df63c372263a4d27877216eb9201f008"
format_stamp: "Formatted at 2026-08-06 04:43:38 on dist-test-slave-vkjp"
server_key: "b5652ef88f67bb64de789338abddfc28"
server_key_iv: "10d1b50b2dd6938eb3068b7fd5e6a985"
server_key_version: "encryptionkey@0"
I20260806 04:43:38.510013 19984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:38.519289 19984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:38.519969 19984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:38.520169 19984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:38.520742 19984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:38.521684 19984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:38.521764 19984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:38.521847 19984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:38.521899 19984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:38.530763 19984 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:45825
I20260806 04:43:38.530786 20119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:45825 every 8 connection(s)
I20260806 04:43:38.531737 19984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:43:38.534420 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 19984
I20260806 04:43:38.534559 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:43:38.534875 32410 external_mini_cluster.cc:1428] Setting key 9f4f04d2a54d914ef452b91281f7d602
I20260806 04:43:38.536929 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33331
--builtin_ntp_servers=127.31.166.148:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:43:38.541726 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:38.533588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:53445 (local address 127.31.166.190:33331)
0806 04:43:38.533732 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:38.533737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:38.534854 (+ 1117us) server_negotiation.cc:407] Connection header received
0806 04:43:38.536512 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:38.536515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:38.536573 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:38.536664 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:38.537668 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.538252 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.539490 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.539912 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.540036 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:38.541095 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:38.541102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:38.541106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:38.541125 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:38.541154 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:38.541168 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:38.541254 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:38.541406 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 04:43:38.541512 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:43:38.542433 20122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:38.533968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33331 (local address 127.31.166.129:53445)
0806 04:43:38.534696 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:38.534737 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:38.536287 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:38.536855 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:38.536861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:38.536943 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:38.537524 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:38.537537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.538416 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.538421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:38.539366 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:38.539376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.540060 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.540062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:38.540205 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:38.540824 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:43:38.540826 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:38.540870 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:43:38.540931 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:38.541268 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:38.541274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:38.541278 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:43:38.541370 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 04:43:38.541587 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":149,"threads_started":1}
I20260806 04:43:38.543745 20120 heartbeater.cc:347] Connected to a master server at 127.31.166.190:33331
I20260806 04:43:38.544001 20120 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:38.544512 20120 heartbeater.cc:511] Master 127.31.166.190:33331 requested a full tablet report, sending...
I20260806 04:43:38.546011 19923 ts_manager.cc:194] Registered new tserver with Master: df63c372263a4d27877216eb9201f008 (127.31.166.129:45825)
I20260806 04:43:38.547248 19923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.166.129:53445
W20260806 04:43:38.651307 20123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:38.651575 20123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:38.651614 20123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:38.655767 20123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:38.655824 20123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:38.655882 20123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:43:38.660667 20123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:38.661731 20123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:38.663223 20123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:43:38.673141 20123 server_base.cc:1034] running on GCE node
W20260806 04:43:38.673192 20129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:38.673308 20130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:38.673390 20132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:38.673874 20123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:38.674525 20123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:38.675711 20123 hybrid_clock.cc:648] HybridClock initialized: now 1785991418675688 us; error 42 us; skew 500 ppm
I20260806 04:43:38.677561 20123 webserver.cc:460] Webserver started at http://127.31.166.130:42857/ using document root <none> and password file <none>
I20260806 04:43:38.678148 20123 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:38.678241 20123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:38.678478 20123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:38.680605 20123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "d5290483d0294f3ea993032685548739"
format_stamp: "Formatted at 2026-08-06 04:43:38 on dist-test-slave-vkjp"
server_key: "93315b58301f5354f3d36067b58bef81"
server_key_iv: "ef60606c8565b4c5eee3a1ef2e05c714"
server_key_version: "encryptionkey@0"
I20260806 04:43:38.681253 20123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "d5290483d0294f3ea993032685548739"
format_stamp: "Formatted at 2026-08-06 04:43:38 on dist-test-slave-vkjp"
server_key: "93315b58301f5354f3d36067b58bef81"
server_key_iv: "ef60606c8565b4c5eee3a1ef2e05c714"
server_key_version: "encryptionkey@0"
I20260806 04:43:38.685022 20123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:43:38.687583 20138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:38.688814 20123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:43:38.688971 20123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "d5290483d0294f3ea993032685548739"
format_stamp: "Formatted at 2026-08-06 04:43:38 on dist-test-slave-vkjp"
server_key: "93315b58301f5354f3d36067b58bef81"
server_key_iv: "ef60606c8565b4c5eee3a1ef2e05c714"
server_key_version: "encryptionkey@0"
I20260806 04:43:38.689090 20123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:38.730881 20123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:38.731808 20123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:38.732057 20123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:38.732749 20123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:38.733845 20123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:38.733923 20123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:38.734036 20123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:38.734083 20123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:38.743863 20123 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:42763
I20260806 04:43:38.743892 20257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:42763 every 8 connection(s)
I20260806 04:43:38.745003 20123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:43:38.754748 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 20123
I20260806 04:43:38.754917 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:43:38.754981 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:38.747059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:38697 (local address 127.31.166.190:33331)
0806 04:43:38.747252 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:38.747256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:38.748444 (+ 1188us) server_negotiation.cc:407] Connection header received
0806 04:43:38.749933 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:38.749936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:38.749988 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:38.750082 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:38.751064 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.751671 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.752883 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.753108 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.753191 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:38.754400 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:38.754408 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:38.754413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:38.754431 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:38.754460 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:38.754474 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:38.754557 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:38.754721 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 04:43:38.754803 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 04:43:38.755307 32410 external_mini_cluster.cc:1428] Setting key b91b71721a35797ed9f94a4d9fa1c5ab
I20260806 04:43:38.756062 20260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:38.747596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33331 (local address 127.31.166.130:38697)
0806 04:43:38.748307 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:38.748352 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:38.749711 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:38.750284 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:38.750291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:38.750376 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:38.750903 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:38.750916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.751811 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.751814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:38.752763 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:38.752773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.753244 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.753247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:38.753425 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:38.754093 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:43:38.754095 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:38.754136 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:43:38.754198 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:38.754572 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:38.754579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:38.754583 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:43:38.754672 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 04:43:38.754932 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":138,"threads_started":1}
I20260806 04:43:38.757272 20258 heartbeater.cc:347] Connected to a master server at 127.31.166.190:33331
I20260806 04:43:38.757525 20258 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:38.757531 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33331
--builtin_ntp_servers=127.31.166.148:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:43:38.758158 20258 heartbeater.cc:511] Master 127.31.166.190:33331 requested a full tablet report, sending...
I20260806 04:43:38.759454 19923 ts_manager.cc:194] Registered new tserver with Master: d5290483d0294f3ea993032685548739 (127.31.166.130:42763)
I20260806 04:43:38.760351 19923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.166.130:38697
W20260806 04:43:38.873855 20261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:38.874179 20261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:38.874267 20261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:38.878407 20261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:38.878468 20261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:38.878523 20261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:43:38.883323 20261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:33331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:38.884434 20261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:38.885803 20261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:38.895862 20266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:38.895920 20267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:38.895927 20269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:38.896341 20261 server_base.cc:1034] running on GCE node
I20260806 04:43:38.896811 20261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:38.897642 20261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:38.898962 20261 hybrid_clock.cc:648] HybridClock initialized: now 1785991418898933 us; error 34 us; skew 500 ppm
I20260806 04:43:38.901283 20261 webserver.cc:460] Webserver started at http://127.31.166.131:46183/ using document root <none> and password file <none>
I20260806 04:43:38.902009 20261 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:38.902122 20261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:38.902398 20261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:38.904811 20261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "cc03f88e0d174f3780a706acd5a2167e"
format_stamp: "Formatted at 2026-08-06 04:43:38 on dist-test-slave-vkjp"
server_key: "8389fd825cb20f6375e8e7b511cc1262"
server_key_iv: "789096be8fe99827cf75ee7282bc90a9"
server_key_version: "encryptionkey@0"
I20260806 04:43:38.905440 20261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "cc03f88e0d174f3780a706acd5a2167e"
format_stamp: "Formatted at 2026-08-06 04:43:38 on dist-test-slave-vkjp"
server_key: "8389fd825cb20f6375e8e7b511cc1262"
server_key_iv: "789096be8fe99827cf75ee7282bc90a9"
server_key_version: "encryptionkey@0"
I20260806 04:43:38.909544 20261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:43:38.912269 20275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:38.913611 20261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:43:38.913741 20261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "cc03f88e0d174f3780a706acd5a2167e"
format_stamp: "Formatted at 2026-08-06 04:43:38 on dist-test-slave-vkjp"
server_key: "8389fd825cb20f6375e8e7b511cc1262"
server_key_iv: "789096be8fe99827cf75ee7282bc90a9"
server_key_version: "encryptionkey@0"
I20260806 04:43:38.913861 20261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:38.937141 20261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:38.937891 20261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:38.938113 20261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:38.938753 20261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:38.939774 20261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:38.939850 20261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:38.939926 20261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:38.939981 20261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:38.952553 20261 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:34167
I20260806 04:43:38.952587 20391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:34167 every 8 connection(s)
I20260806 04:43:38.953862 20261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:43:38.955344 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 20261
I20260806 04:43:38.955473 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:43:38.955822 32410 external_mini_cluster.cc:1428] Setting key a9a3d7a8769825495fc2cd9f3be63848
I20260806 04:43:38.966554 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:38.956158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:54105 (local address 127.31.166.190:33331)
0806 04:43:38.956324 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:38.956329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:38.957862 (+ 1533us) server_negotiation.cc:407] Connection header received
0806 04:43:38.959939 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:38.959942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:38.960034 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:38.960166 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:38.961616 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.962438 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.964148 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.964506 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.964655 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:38.965979 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:38.965982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:38.965986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:38.966002 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:38.966032 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:38.966041 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:38.966127 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:38.966315 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 04:43:38.966389 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:43:38.967844 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:38.956755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:33331 (local address 127.31.166.131:54105)
0806 04:43:38.957696 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:38.957759 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:38.959671 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:38.960403 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:38.960419 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:38.960569 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:38.961451 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:38.961473 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.962584 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.962588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:38.963944 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:38.963956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.964727 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.964734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:38.964932 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:38.965660 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:43:38.965662 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:38.965708 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:43:38.965782 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:38.966144 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:38.966155 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:38.966161 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 04:43:38.966315 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 04:43:38.966639 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":143,"threads_started":1}
I20260806 04:43:38.969295 20392 heartbeater.cc:347] Connected to a master server at 127.31.166.190:33331
I20260806 04:43:38.970045 20392 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:38.970705 20392 heartbeater.cc:511] Master 127.31.166.190:33331 requested a full tablet report, sending...
I20260806 04:43:38.972013 19923 ts_manager.cc:194] Registered new tserver with Master: cc03f88e0d174f3780a706acd5a2167e (127.31.166.131:34167)
I20260806 04:43:38.972862 19923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.166.131:54105
I20260806 04:43:38.982427 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:43:38.990677 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:38.985336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.31.166.190:33331)
0806 04:43:38.985497 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:38.985503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:38.985766 (+ 263us) server_negotiation.cc:407] Connection header received
0806 04:43:38.985927 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:38.985930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:38.985978 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:38.986046 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:38.986461 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.987053 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.988548 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:38.988786 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:38.988880 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:38.989938 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:38.989941 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:38.989943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:38.989957 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:38.989977 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:38.989985 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:38.990096 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:38.990415 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 04:43:38.990467 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:43:38.994925 19923 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:43:38.998277 19923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:43:39.014983 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:39.007981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:45825 (local address 127.0.0.1:58508)
0806 04:43:39.009262 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:39.009290 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:39.009371 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:39.009945 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:39.009951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:39.009976 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:39.010136 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:39.010155 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.011834 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.011846 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:39.013299 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:39.013317 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.014349 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.014353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:39.014473 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:39.014517 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 04:43:39.014627 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 04:43:39.014765 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1204,"mutex_wait_us":177,"thread_start_us":127,"threads_started":1}
I20260806 04:43:39.015020 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:39.008859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58508 (local address 127.31.166.129:45825)
0806 04:43:39.009198 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:39.009205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:39.009390 (+ 185us) server_negotiation.cc:407] Connection header received
0806 04:43:39.009521 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:39.009526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:39.009746 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:39.010292 (+ 546us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:39.010435 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.011699 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.013430 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.014211 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.014453 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:39.014656 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:39.014710 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 04:43:39.014820 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":118,"threads_started":1}
I20260806 04:43:39.015640 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:39.008852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42142 (local address 127.31.166.130:42763)
0806 04:43:39.009583 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:39.009595 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:39.009905 (+ 310us) server_negotiation.cc:407] Connection header received
0806 04:43:39.010045 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:39.010050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:39.010246 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:39.010419 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:39.010762 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.011667 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.013744 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.014507 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.014623 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:39.014760 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:39.015386 (+ 626us) server_negotiation.cc:299] Negotiation successful
0806 04:43:39.015485 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":165,"threads_started":1}
I20260806 04:43:39.016423 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:39.008455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49166 (local address 127.31.166.131:34167)
0806 04:43:39.009773 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:39.009784 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:39.009809 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:43:39.009889 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:39.009897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:39.010073 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:39.010252 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:39.010494 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.011696 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.014044 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.014811 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.014937 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:39.016137 (+ 1200us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:39.016189 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 04:43:39.016290 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1218,"thread_start_us":117,"threads_started":1}
I20260806 04:43:39.015960 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:39.008688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:42763 (local address 127.0.0.1:42142)
0806 04:43:39.009792 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:39.009804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:39.009919 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:39.010536 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:39.010540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:39.010560 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:39.010642 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:39.010647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.011878 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.011885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:39.013608 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:39.013620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.014824 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.014827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:39.014939 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:39.014961 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 04:43:39.015038 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 04:43:39.015815 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":108,"thread_start_us":100,"threads_started":1}
I20260806 04:43:39.016954 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:39.007879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:34167 (local address 127.0.0.1:49166)
0806 04:43:39.008906 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:39.008968 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:39.009110 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:39.010237 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:39.010240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:39.010258 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:39.010397 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:39.010404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.011834 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.011838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:39.013893 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:39.013906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.015531 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.015535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:39.015628 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:39.015656 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 04:43:39.015936 (+ 280us) client_negotiation.cc:240] Negotiation successful
0806 04:43:39.016023 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":109,"thread_start_us":117,"threads_started":1}
I20260806 04:43:39.018400 20050 tablet_service.cc:1467] Processing CreateTablet for tablet e210a43af52c4d83a91146158828f7a8 (DEFAULT_TABLE table=test-table [id=38bd44a85d2545bb9e382a1495ebfb2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:39.019723 20050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e210a43af52c4d83a91146158828f7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:39.019955 20188 tablet_service.cc:1467] Processing CreateTablet for tablet e210a43af52c4d83a91146158828f7a8 (DEFAULT_TABLE table=test-table [id=38bd44a85d2545bb9e382a1495ebfb2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:39.020131 20323 tablet_service.cc:1467] Processing CreateTablet for tablet e210a43af52c4d83a91146158828f7a8 (DEFAULT_TABLE table=test-table [id=38bd44a85d2545bb9e382a1495ebfb2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:39.021013 20188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e210a43af52c4d83a91146158828f7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:39.021190 20323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e210a43af52c4d83a91146158828f7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:39.027401 20410 tablet_bootstrap.cc:492] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e: Bootstrap starting.
I20260806 04:43:39.028779 20411 tablet_bootstrap.cc:492] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008: Bootstrap starting.
I20260806 04:43:39.029028 20412 tablet_bootstrap.cc:492] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739: Bootstrap starting.
I20260806 04:43:39.030138 20410 tablet_bootstrap.cc:654] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:39.030732 20411 tablet_bootstrap.cc:654] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:39.031245 20412 tablet_bootstrap.cc:654] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:39.031595 20411 log.cc:824] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:39.031633 20410 log.cc:824] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:39.032078 20412 log.cc:824] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:39.033820 20411 tablet_bootstrap.cc:492] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008: No bootstrap required, opened a new log
I20260806 04:43:39.033819 20410 tablet_bootstrap.cc:492] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e: No bootstrap required, opened a new log
I20260806 04:43:39.033885 20412 tablet_bootstrap.cc:492] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739: No bootstrap required, opened a new log
I20260806 04:43:39.034025 20410 ts_tablet_manager.cc:1397] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260806 04:43:39.034049 20411 ts_tablet_manager.cc:1397] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260806 04:43:39.034125 20412 ts_tablet_manager.cc:1397] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:43:39.037712 20412 raft_consensus.cc:348] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc03f88e0d174f3780a706acd5a2167e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34167 } } peers { permanent_uuid: "df63c372263a4d27877216eb9201f008" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45825 } } peers { permanent_uuid: "d5290483d0294f3ea993032685548739" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42763 } }
I20260806 04:43:39.038026 20412 raft_consensus.cc:374] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:39.038085 20412 raft_consensus.cc:729] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5290483d0294f3ea993032685548739, State: Initialized, Role: FOLLOWER
I20260806 04:43:39.038347 20411 raft_consensus.cc:348] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc03f88e0d174f3780a706acd5a2167e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34167 } } peers { permanent_uuid: "df63c372263a4d27877216eb9201f008" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45825 } } peers { permanent_uuid: "d5290483d0294f3ea993032685548739" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42763 } }
I20260806 04:43:39.038340 20410 raft_consensus.cc:348] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc03f88e0d174f3780a706acd5a2167e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34167 } } peers { permanent_uuid: "df63c372263a4d27877216eb9201f008" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45825 } } peers { permanent_uuid: "d5290483d0294f3ea993032685548739" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42763 } }
I20260806 04:43:39.038589 20410 raft_consensus.cc:374] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:39.038590 20411 raft_consensus.cc:374] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:39.038641 20411 raft_consensus.cc:729] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df63c372263a4d27877216eb9201f008, State: Initialized, Role: FOLLOWER
I20260806 04:43:39.038641 20410 raft_consensus.cc:729] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc03f88e0d174f3780a706acd5a2167e, State: Initialized, Role: FOLLOWER
I20260806 04:43:39.038919 20412 consensus_queue.cc:260] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc03f88e0d174f3780a706acd5a2167e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34167 } } peers { permanent_uuid: "df63c372263a4d27877216eb9201f008" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45825 } } peers { permanent_uuid: "d5290483d0294f3ea993032685548739" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42763 } }
I20260806 04:43:39.039068 20410 consensus_queue.cc:260] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc03f88e0d174f3780a706acd5a2167e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34167 } } peers { permanent_uuid: "df63c372263a4d27877216eb9201f008" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45825 } } peers { permanent_uuid: "d5290483d0294f3ea993032685548739" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42763 } }
I20260806 04:43:39.039068 20411 consensus_queue.cc:260] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc03f88e0d174f3780a706acd5a2167e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34167 } } peers { permanent_uuid: "df63c372263a4d27877216eb9201f008" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45825 } } peers { permanent_uuid: "d5290483d0294f3ea993032685548739" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42763 } }
I20260806 04:43:39.039726 20258 heartbeater.cc:503] Master 127.31.166.190:33331 was elected leader, sending a full tablet report...
I20260806 04:43:39.040114 20411 ts_tablet_manager.cc:1428] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260806 04:43:39.040114 20410 ts_tablet_manager.cc:1428] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260806 04:43:39.040113 20412 ts_tablet_manager.cc:1428] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260806 04:43:39.040442 20120 heartbeater.cc:503] Master 127.31.166.190:33331 was elected leader, sending a full tablet report...
I20260806 04:43:39.040442 20392 heartbeater.cc:503] Master 127.31.166.190:33331 was elected leader, sending a full tablet report...
I20260806 04:43:39.046271 20418 raft_consensus.cc:482] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:43:39.046502 20418 raft_consensus.cc:504] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc03f88e0d174f3780a706acd5a2167e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34167 } } peers { permanent_uuid: "df63c372263a4d27877216eb9201f008" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45825 } } peers { permanent_uuid: "d5290483d0294f3ea993032685548739" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42763 } }
I20260806 04:43:39.047735 20418 leader_election.cc:288] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc03f88e0d174f3780a706acd5a2167e (127.31.166.131:34167), d5290483d0294f3ea993032685548739 (127.31.166.130:42763)
I20260806 04:43:39.052004 20420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:39.047971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:42763 (local address 127.31.166.129:58413)
0806 04:43:39.048563 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:39.048576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:39.048675 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:39.048923 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:39.048926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:39.048939 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:39.049076 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:39.049083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.049794 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.049799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:39.050742 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:39.050751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.051543 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.051547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:39.051671 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:39.051704 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 04:43:39.051795 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 04:43:39.051876 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":23,"thread_start_us":98,"threads_started":1}
I20260806 04:43:39.052094 20409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:39.048050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:58413 (local address 127.31.166.130:42763)
0806 04:43:39.048190 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:39.048195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:39.048658 (+ 463us) server_negotiation.cc:407] Connection header received
0806 04:43:39.048778 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:39.048781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:39.048829 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:39.048905 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:39.049190 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.049669 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.050875 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.051422 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.051516 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:39.051553 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:39.051860 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 04:43:39.051945 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 04:43:39.052338 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:39.047941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:34167 (local address 127.31.166.129:55133)
0806 04:43:39.048212 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:39.048228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:39.048350 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:39.048729 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:39.048732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:39.048760 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:39.048942 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:39.048950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.049805 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.049808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:39.050860 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:39.050872 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.051846 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.051850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:39.051952 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:39.051975 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 04:43:39.052063 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 04:43:39.052136 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":65,"thread_start_us":117,"threads_started":1}
I20260806 04:43:39.052358 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:39.048335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:55133 (local address 127.31.166.131:34167)
0806 04:43:39.048479 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:39.048485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:39.048499 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:43:39.048548 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:39.048551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:39.048614 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:39.048700 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:39.049064 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.049672 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.050988 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.051590 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.051720 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:39.051789 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:39.052103 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 04:43:39.052208 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:43:39.052894 20210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e210a43af52c4d83a91146158828f7a8" candidate_uuid: "df63c372263a4d27877216eb9201f008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5290483d0294f3ea993032685548739" is_pre_election: true
I20260806 04:43:39.053084 20344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e210a43af52c4d83a91146158828f7a8" candidate_uuid: "df63c372263a4d27877216eb9201f008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc03f88e0d174f3780a706acd5a2167e" is_pre_election: true
I20260806 04:43:39.053205 20210 raft_consensus.cc:2454] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df63c372263a4d27877216eb9201f008 in term 0.
I20260806 04:43:39.053395 20344 raft_consensus.cc:2454] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df63c372263a4d27877216eb9201f008 in term 0.
I20260806 04:43:39.053784 19999 leader_election.cc:302] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5290483d0294f3ea993032685548739, df63c372263a4d27877216eb9201f008; no voters:
I20260806 04:43:39.054105 20418 raft_consensus.cc:2781] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:43:39.054186 20418 raft_consensus.cc:482] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:43:39.054217 20418 raft_consensus.cc:3037] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:39.055298 20418 raft_consensus.cc:504] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc03f88e0d174f3780a706acd5a2167e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34167 } } peers { permanent_uuid: "df63c372263a4d27877216eb9201f008" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45825 } } peers { permanent_uuid: "d5290483d0294f3ea993032685548739" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42763 } }
I20260806 04:43:39.055739 20418 leader_election.cc:288] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [CANDIDATE]: Term 1 election: Requested vote from peers cc03f88e0d174f3780a706acd5a2167e (127.31.166.131:34167), d5290483d0294f3ea993032685548739 (127.31.166.130:42763)
I20260806 04:43:39.055974 20344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e210a43af52c4d83a91146158828f7a8" candidate_uuid: "df63c372263a4d27877216eb9201f008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc03f88e0d174f3780a706acd5a2167e"
I20260806 04:43:39.055974 20210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e210a43af52c4d83a91146158828f7a8" candidate_uuid: "df63c372263a4d27877216eb9201f008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5290483d0294f3ea993032685548739"
I20260806 04:43:39.056098 20344 raft_consensus.cc:3037] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:39.056098 20210 raft_consensus.cc:3037] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:39.057381 20344 raft_consensus.cc:2454] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df63c372263a4d27877216eb9201f008 in term 1.
I20260806 04:43:39.057395 20210 raft_consensus.cc:2454] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df63c372263a4d27877216eb9201f008 in term 1.
I20260806 04:43:39.057684 19999 leader_election.cc:302] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5290483d0294f3ea993032685548739, df63c372263a4d27877216eb9201f008; no voters:
I20260806 04:43:39.057925 20418 raft_consensus.cc:2781] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:39.058401 20418 raft_consensus.cc:686] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [term 1 LEADER]: Becoming Leader. State: Replica: df63c372263a4d27877216eb9201f008, State: Running, Role: LEADER
I20260806 04:43:39.058821 20418 consensus_queue.cc:237] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc03f88e0d174f3780a706acd5a2167e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34167 } } peers { permanent_uuid: "df63c372263a4d27877216eb9201f008" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45825 } } peers { permanent_uuid: "d5290483d0294f3ea993032685548739" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42763 } }
I20260806 04:43:39.062367 19922 catalog_manager.cc:5495] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 reported cstate change: term changed from 0 to 1, leader changed from <none> to df63c372263a4d27877216eb9201f008 (127.31.166.129). New cstate: current_term: 1 leader_uuid: "df63c372263a4d27877216eb9201f008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc03f88e0d174f3780a706acd5a2167e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df63c372263a4d27877216eb9201f008" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 45825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5290483d0294f3ea993032685548739" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42763 } health_report { overall_health: UNKNOWN } } }
I20260806 04:43:39.079665 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:39.075902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58514 (local address 127.31.166.129:45825)
0806 04:43:39.076054 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:39.076059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:39.076074 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:43:39.076221 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:39.076223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:39.076281 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:39.076417 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:43:39.076670 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.077173 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.078057 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:39.078264 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:39.078369 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:39.078698 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:43:39.079198 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:43:39.079367 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:39.079469 (+ 102us) server_negotiation.cc:299] Negotiation successful
0806 04:43:39.079510 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 04:43:39.085776 20344 raft_consensus.cc:1261] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e [term 1 FOLLOWER]: Refusing update from remote peer df63c372263a4d27877216eb9201f008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:43:39.085776 20210 raft_consensus.cc:1261] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739 [term 1 FOLLOWER]: Refusing update from remote peer df63c372263a4d27877216eb9201f008: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:43:39.086514 20421 consensus_queue.cc:1035] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc03f88e0d174f3780a706acd5a2167e" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 34167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:43:39.086764 20418 consensus_queue.cc:1035] T e210a43af52c4d83a91146158828f7a8 P df63c372263a4d27877216eb9201f008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5290483d0294f3ea993032685548739" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 42763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:43:39.103853 19922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:43:39.104060 19922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:43:39.106770 19922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8dccb1ba8e654ce9833e7630d5bd3cbb: Sending DeleteTablet for 3 replicas of tablet e210a43af52c4d83a91146158828f7a8
I20260806 04:43:39.107676 20323 tablet_service.cc:1514] Processing DeleteTablet for tablet e210a43af52c4d83a91146158828f7a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:39 UTC) from {username='slave'} at 127.0.0.1:49166
I20260806 04:43:39.107791 20050 tablet_service.cc:1514] Processing DeleteTablet for tablet e210a43af52c4d83a91146158828f7a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:39 UTC) from {username='slave'} at 127.0.0.1:58508
I20260806 04:43:39.107812 20188 tablet_service.cc:1514] Processing DeleteTablet for tablet e210a43af52c4d83a91146158828f7a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:39 UTC) from {username='slave'} at 127.0.0.1:42142
I20260806 04:43:39.108258 20434 tablet_replica.cc:330] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e: stopping tablet replica
I20260806 04:43:39.108305 20435 tablet_replica.cc:330] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739: stopping tablet replica
I20260806 04:43:39.108672 20434 raft_consensus.cc:2229] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:43:39.108673 20435 raft_consensus.cc:2229] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:43:39.109018 20435 raft_consensus.cc:2258] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:43:39.109081 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 19984
I20260806 04:43:39.109053 20434 raft_consensus.cc:2258] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:43:39.112098 20434 ts_tablet_manager.cc:1905] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:43:39.113679 20435 ts_tablet_manager.cc:1905] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:43:39.114928 20434 ts_tablet_manager.cc:1918] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:43:39.115039 20434 log.cc:1196] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/e210a43af52c4d83a91146158828f7a8
I20260806 04:43:39.115401 20434 ts_tablet_manager.cc:1939] T e210a43af52c4d83a91146158828f7a8 P cc03f88e0d174f3780a706acd5a2167e: Deleting consensus metadata
I20260806 04:43:39.116649 19907 catalog_manager.cc:4841] TS cc03f88e0d174f3780a706acd5a2167e (127.31.166.131:34167): tablet e210a43af52c4d83a91146158828f7a8 (table test-table [id=38bd44a85d2545bb9e382a1495ebfb2b]) successfully deleted
I20260806 04:43:39.117144 20435 ts_tablet_manager.cc:1918] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:43:39.117273 20435 log.cc:1196] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/e210a43af52c4d83a91146158828f7a8
I20260806 04:43:39.117609 20435 ts_tablet_manager.cc:1939] T e210a43af52c4d83a91146158828f7a8 P d5290483d0294f3ea993032685548739: Deleting consensus metadata
W20260806 04:43:39.118858 19906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:45825 (error 108)
I20260806 04:43:39.119031 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 20123
I20260806 04:43:39.120812 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:39.120401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:45825 (local address 127.0.0.1:58518)
0806 04:43:39.120584 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:39.120692 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:45825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 04:43:39.121937 19906 catalog_manager.cc:4584] TS df63c372263a4d27877216eb9201f008 (127.31.166.129:45825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e210a43af52c4d83a91146158828f7a8: Network error: Client connection negotiation failed: client connection to 127.31.166.129:45825: connect: Connection refused (error 111)
I20260806 04:43:39.126995 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 20261
I20260806 04:43:39.128417 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:39.127163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:42763 (local address 127.0.0.1:42146)
0806 04:43:39.128216 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:39.128289 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.130:42763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":942}
I20260806 04:43:39.133031 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 19890
2026-08-06T04:43:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1045 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:39 dist-test-slave-vkjp krb5kdc[20440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:39 dist-test-slave-vkjp krb5kdc[20440](info): set up 2 sockets
Aug 06 04:43:39 dist-test-slave-vkjp krb5kdc[20440](info): commencing operation
krb5kdc: starting...
W20260806 04:43:40.205499 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:43:40 dist-test-slave-vkjp krb5kdc[20440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:43:40.334648 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:36141
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:36141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:40.448761 20454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:40.449074 20454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:40.449157 20454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:40.453181 20454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:40.453281 20454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:40.453332 20454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:40.453392 20454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:40.453433 20454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:40.458765 20454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:36141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:36141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:40.459937 20454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:40.460980 20454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:40.470110 20462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:40.470126 20460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:40.470160 20454 server_base.cc:1034] running on GCE node
W20260806 04:43:40.470368 20459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:40.470942 20454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:40.472072 20454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:40.473268 20454 hybrid_clock.cc:648] HybridClock initialized: now 1785991420473256 us; error 36 us; skew 500 ppm
Aug 06 04:43:40 dist-test-slave-vkjp krb5kdc[20440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991420, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:40.483569 20454 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:43:40.484759 20454 webserver.cc:460] Webserver started at http://127.31.166.190:37217/ using document root <none> and password file <none>
I20260806 04:43:40.485421 20454 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:40.485532 20454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:40.485822 20454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:40.488350 20454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "f68c1aebf5f8406a81795eddac5f76b9"
format_stamp: "Formatted at 2026-08-06 04:43:40 on dist-test-slave-vkjp"
server_key: "2b5e7f6e2a275d005f13592838dfce26"
server_key_iv: "1e273b62ba1b0727791c8e14391cc7e0"
server_key_version: "encryptionkey@0"
I20260806 04:43:40.489066 20454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "f68c1aebf5f8406a81795eddac5f76b9"
format_stamp: "Formatted at 2026-08-06 04:43:40 on dist-test-slave-vkjp"
server_key: "2b5e7f6e2a275d005f13592838dfce26"
server_key_iv: "1e273b62ba1b0727791c8e14391cc7e0"
server_key_version: "encryptionkey@0"
I20260806 04:43:40.493418 20454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:43:40.496146 20470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:40.497514 20454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:43:40.497681 20454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "f68c1aebf5f8406a81795eddac5f76b9"
format_stamp: "Formatted at 2026-08-06 04:43:40 on dist-test-slave-vkjp"
server_key: "2b5e7f6e2a275d005f13592838dfce26"
server_key_iv: "1e273b62ba1b0727791c8e14391cc7e0"
server_key_version: "encryptionkey@0"
I20260806 04:43:40.497823 20454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:40.520305 20454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:40.523483 20454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:40.523689 20454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:40.530961 20454 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:36141
I20260806 04:43:40.530997 20524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:36141 every 8 connection(s)
I20260806 04:43:40.531945 20454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:40.534576 20525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:40.539970 20525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9: Bootstrap starting.
I20260806 04:43:40.541987 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 20454
I20260806 04:43:40.542141 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:40.542433 32410 external_mini_cluster.cc:1428] Setting key 01745544000d772a7539730212f5e40c
I20260806 04:43:40.542574 20525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:40.543423 20525 log.cc:824] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:40.545476 20525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9: No bootstrap required, opened a new log
I20260806 04:43:40.549537 20525 raft_consensus.cc:348] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68c1aebf5f8406a81795eddac5f76b9" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36141 } }
I20260806 04:43:40.549821 20525 raft_consensus.cc:374] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:40.549894 20525 raft_consensus.cc:729] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68c1aebf5f8406a81795eddac5f76b9, State: Initialized, Role: FOLLOWER
I20260806 04:43:40.550405 20525 consensus_queue.cc:260] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68c1aebf5f8406a81795eddac5f76b9" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36141 } }
I20260806 04:43:40.550572 20525 raft_consensus.cc:388] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:40.550644 20525 raft_consensus.cc:482] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:40.550756 20525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [term 0 FOLLOWER]: Advancing to term 1
Aug 06 04:43:40 dist-test-slave-vkjp krb5kdc[20440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:43:40.551795 20525 raft_consensus.cc:504] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68c1aebf5f8406a81795eddac5f76b9" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36141 } }
I20260806 04:43:40.552122 20525 leader_election.cc:302] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f68c1aebf5f8406a81795eddac5f76b9; no voters:
I20260806 04:43:40.552431 20525 leader_election.cc:288] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:40.552611 20531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:40.552838 20531 raft_consensus.cc:686] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [term 1 LEADER]: Becoming Leader. State: Replica: f68c1aebf5f8406a81795eddac5f76b9, State: Running, Role: LEADER
I20260806 04:43:40.553171 20531 consensus_queue.cc:237] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68c1aebf5f8406a81795eddac5f76b9" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36141 } }
I20260806 04:43:40.553829 20525 sys_catalog.cc:564] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:40.555572 20532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f68c1aebf5f8406a81795eddac5f76b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68c1aebf5f8406a81795eddac5f76b9" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36141 } } }
I20260806 04:43:40.555706 20532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:40.556001 20533 sys_catalog.cc:455] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f68c1aebf5f8406a81795eddac5f76b9. Latest consensus state: current_term: 1 leader_uuid: "f68c1aebf5f8406a81795eddac5f76b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f68c1aebf5f8406a81795eddac5f76b9" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 36141 } } }
I20260806 04:43:40.557016 20533 sys_catalog.cc:458] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:40.558398 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:40.543647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.31.166.190:36141)
0806 04:43:40.544082 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:40.544096 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:40.544131 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:43:40.544736 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:40.544751 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:40.545047 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:40.545318 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:40.546230 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:40.547279 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:40.548516 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:40.548988 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:40.549101 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:40.551941 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:40.551948 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:40.551966 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:40.551997 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:40.554051 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:40.554569 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:40.554575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:40.554581 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:40.554724 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:40.555059 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:40.555063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:40.555064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:40.555682 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:40.555912 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:40.557303 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0806 04:43:40.557643 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":140,"threads_started":1}
I20260806 04:43:40.558696 20546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:40.561213 20546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:40.566649 20546 catalog_manager.cc:1347] Generated new cluster ID: 620f8b544030489db2c8a457257e135d
I20260806 04:43:40.566712 20546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:40.579155 20546 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:40.580417 20546 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:40.594385 20546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9: Generated new TSK 0
I20260806 04:43:40.595311 20546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.166.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
Entry for principal kudu/127.31.166.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab.
I20260806 04:43:40.634361 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36141
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:40.749236 20556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:40.749517 20556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:40.749562 20556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:40.753863 20556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:40.753929 20556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:40.754019 20556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:43:40.759922 20556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.129.keytab
--principal=kudu/127.31.166.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:40.761080 20556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:40.762130 20556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:40.771664 20565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:40.771659 20562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:40.771713 20556 server_base.cc:1034] running on GCE node
W20260806 04:43:40.771659 20563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:40.772269 20556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:40.772957 20556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:40.774175 20556 hybrid_clock.cc:648] HybridClock initialized: now 1785991420774155 us; error 30 us; skew 500 ppm
Aug 06 04:43:40 dist-test-slave-vkjp krb5kdc[20440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991420, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:40.777040 20556 init.cc:377] Logged in from keytab as kudu/127.31.166.129@KRBTEST.COM (short username kudu)
I20260806 04:43:40.777987 20556 webserver.cc:460] Webserver started at http://127.31.166.129:40533/ using document root <none> and password file <none>
I20260806 04:43:40.778590 20556 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:40.778684 20556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:40.778919 20556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:40.781077 20556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "60e99364e33d46b6acc1aecc66dfe164"
format_stamp: "Formatted at 2026-08-06 04:43:40 on dist-test-slave-vkjp"
server_key: "4d80fe6d023e31a8bd4377b90c65c56f"
server_key_iv: "ae27447a656542400a04e625557330d2"
server_key_version: "encryptionkey@0"
I20260806 04:43:40.781646 20556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "60e99364e33d46b6acc1aecc66dfe164"
format_stamp: "Formatted at 2026-08-06 04:43:40 on dist-test-slave-vkjp"
server_key: "4d80fe6d023e31a8bd4377b90c65c56f"
server_key_iv: "ae27447a656542400a04e625557330d2"
server_key_version: "encryptionkey@0"
I20260806 04:43:40.785447 20556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 04:43:40.788038 20573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:40.789268 20556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:43:40.789434 20556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "60e99364e33d46b6acc1aecc66dfe164"
format_stamp: "Formatted at 2026-08-06 04:43:40 on dist-test-slave-vkjp"
server_key: "4d80fe6d023e31a8bd4377b90c65c56f"
server_key_iv: "ae27447a656542400a04e625557330d2"
server_key_version: "encryptionkey@0"
I20260806 04:43:40.789558 20556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:40.804210 20556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:40.807516 20556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:40.807725 20556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:40.808308 20556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:40.809131 20556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:40.809201 20556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:40.809273 20556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:40.809324 20556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:40.818254 20556 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:34653
I20260806 04:43:40.818365 20693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:34653 every 8 connection(s)
I20260806 04:43:40.819254 20556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:43:40.821491 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 20556
I20260806 04:43:40.821637 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:43:40.821987 32410 external_mini_cluster.cc:1428] Setting key 67aad44728141b8297695d93264fef45
Aug 06 04:43:40 dist-test-slave-vkjp krb5kdc[20440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991420, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.129@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.166.130@KRBTEST.COM; defaulting to no policy
I20260806 04:43:40.834725 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:40.821081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:38069 (local address 127.31.166.190:36141)
0806 04:43:40.821232 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:40.821239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:40.822118 (+ 879us) server_negotiation.cc:407] Connection header received
0806 04:43:40.823591 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:40.823594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:40.823635 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:40.823702 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:40.825331 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:40.826251 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:40.827478 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:40.827936 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:40.828067 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:40.830686 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:40.830689 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:40.830691 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:40.830719 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:40.832333 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:40.832968 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:40.832975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:40.832979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:40.833074 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:40.833489 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:40.833493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:40.833495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:40.833700 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:40.833824 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:40.834384 (+ 560us) server_negotiation.cc:299] Negotiation successful
0806 04:43:40.834525 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.130@KRBTEST.COM" created.
I20260806 04:43:40.835773 20696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:40.821396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36141 (local address 127.31.166.129:38069)
0806 04:43:40.821948 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:40.822006 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:40.823363 (+ 1357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:40.823917 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:40.823928 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:40.824451 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:40.825157 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:40.825179 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:40.826392 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:40.826395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:40.827355 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:40.827365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:40.828091 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:40.828094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:40.828239 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:40.828871 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:43:40.828888 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:40.830489 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:40.832488 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:40.832495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:40.832510 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:40.832846 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:40.833179 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:40.833185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:40.833190 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:40.833368 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:40.833866 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:40.833873 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:40.834126 (+ 253us) client_negotiation.cc:769] Sending connection context
0806 04:43:40.834339 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 04:43:40.834686 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":132,"threads_started":1}
I20260806 04:43:40.837183 20694 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36141
I20260806 04:43:40.837446 20694 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:40.837984 20694 heartbeater.cc:511] Master 127.31.166.190:36141 requested a full tablet report, sending...
I20260806 04:43:40.839684 20488 ts_manager.cc:194] Registered new tserver with Master: 60e99364e33d46b6acc1aecc66dfe164 (127.31.166.129:34653)
I20260806 04:43:40.840838 20488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.129@KRBTEST.COM'} at 127.31.166.129:38069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
Entry for principal kudu/127.31.166.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab.
I20260806 04:43:40.854257 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36141
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:40.966234 20699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:40.966495 20699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:40.966539 20699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:40.970937 20699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:40.970993 20699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:40.971083 20699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:43:40.976312 20699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.130.keytab
--principal=kudu/127.31.166.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:40.977308 20699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:40.978591 20699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:40.988253 20706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:40.988289 20708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:40.988610 20699 server_base.cc:1034] running on GCE node
W20260806 04:43:40.988683 20705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:40.989041 20699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:40.989758 20699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:40.990929 20699 hybrid_clock.cc:648] HybridClock initialized: now 1785991420990911 us; error 30 us; skew 500 ppm
Aug 06 04:43:40 dist-test-slave-vkjp krb5kdc[20440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991420, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:40.993964 20699 init.cc:377] Logged in from keytab as kudu/127.31.166.130@KRBTEST.COM (short username kudu)
I20260806 04:43:40.994843 20699 webserver.cc:460] Webserver started at http://127.31.166.130:45111/ using document root <none> and password file <none>
I20260806 04:43:40.995442 20699 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:40.995503 20699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:40.995671 20699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:40.997634 20699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "d6a8e23564db4bcab3cacc55d57c6693"
format_stamp: "Formatted at 2026-08-06 04:43:40 on dist-test-slave-vkjp"
server_key: "0f8ab1972dcab027f7a841981485cc48"
server_key_iv: "246ea813e87e1a39dffb350cf4484263"
server_key_version: "encryptionkey@0"
I20260806 04:43:40.998137 20699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "d6a8e23564db4bcab3cacc55d57c6693"
format_stamp: "Formatted at 2026-08-06 04:43:40 on dist-test-slave-vkjp"
server_key: "0f8ab1972dcab027f7a841981485cc48"
server_key_iv: "246ea813e87e1a39dffb350cf4484263"
server_key_version: "encryptionkey@0"
I20260806 04:43:41.002095 20699 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260806 04:43:41.004637 20716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:41.005959 20699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:43:41.006088 20699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "d6a8e23564db4bcab3cacc55d57c6693"
format_stamp: "Formatted at 2026-08-06 04:43:40 on dist-test-slave-vkjp"
server_key: "0f8ab1972dcab027f7a841981485cc48"
server_key_iv: "246ea813e87e1a39dffb350cf4484263"
server_key_version: "encryptionkey@0"
I20260806 04:43:41.006199 20699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:41.020246 20699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:41.023643 20699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:41.023892 20699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:41.024554 20699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:41.025475 20699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:41.025527 20699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:41.025604 20699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:41.025646 20699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:41.035614 20699 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:45281
I20260806 04:43:41.035636 20833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:45281 every 8 connection(s)
I20260806 04:43:41.036589 20699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:43:41.041232 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 20699
I20260806 04:43:41.041412 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:43:41.041734 32410 external_mini_cluster.cc:1428] Setting key 25a09bbd07e09a0ddd826bb23eafe662
Aug 06 04:43:41 dist-test-slave-vkjp krb5kdc[20440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991420, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.130@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.166.131@KRBTEST.COM; defaulting to no policy
I20260806 04:43:41.052215 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.038529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:40007 (local address 127.31.166.190:36141)
0806 04:43:41.038686 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:41.038690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:41.039611 (+ 921us) server_negotiation.cc:407] Connection header received
0806 04:43:41.040845 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:41.040848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:41.040935 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:41.041063 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:41.042456 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.043068 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.044856 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.045100 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.045218 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.048139 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:41.048142 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:41.048144 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:41.048168 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:41.049737 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:41.050406 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:41.050409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:41.050410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:41.050487 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:41.050886 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:41.050889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:41.050890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:41.051067 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:41.051200 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:41.051893 (+ 693us) server_negotiation.cc:299] Negotiation successful
0806 04:43:41.052081 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 04:43:41.053141 20836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.038919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36141 (local address 127.31.166.130:40007)
0806 04:43:41.039460 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:41.039503 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:41.040643 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:41.041204 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:41.041210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:41.041694 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:41.042290 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.042313 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.043251 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.043254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.044724 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.044738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.045260 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.045264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.045460 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.046072 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:43:41.046088 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:41.047960 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:41.049859 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:41.049866 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:41.049884 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:41.050271 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:41.050581 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:41.050585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:41.050587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:41.050785 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:41.051227 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:41.051237 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:41.051592 (+ 355us) client_negotiation.cc:769] Sending connection context
0806 04:43:41.051816 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 04:43:41.052217 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":182,"threads_started":1}
I20260806 04:43:41.054450 20834 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.131@KRBTEST.COM" created.
I20260806 04:43:41.054755 20834 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:41.055464 20834 heartbeater.cc:511] Master 127.31.166.190:36141 requested a full tablet report, sending...
I20260806 04:43:41.056643 20488 ts_manager.cc:194] Registered new tserver with Master: d6a8e23564db4bcab3cacc55d57c6693 (127.31.166.130:45281)
I20260806 04:43:41.057229 20488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.130@KRBTEST.COM'} at 127.31.166.130:40007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
Entry for principal kudu/127.31.166.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab.
I20260806 04:43:41.070053 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36141
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:41.184219 20839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:41.184636 20839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:41.184703 20839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:41.188911 20839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:41.188974 20839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:41.189033 20839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:43:41.194561 20839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.131.keytab
--principal=kudu/127.31.166.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:36141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:41.195641 20839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:41.196717 20839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:41.206472 20848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:41.206472 20845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:41.206535 20846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:41.206700 20839 server_base.cc:1034] running on GCE node
I20260806 04:43:41.207259 20839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:41.207964 20839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:41.209185 20839 hybrid_clock.cc:648] HybridClock initialized: now 1785991421209166 us; error 34 us; skew 500 ppm
Aug 06 04:43:41 dist-test-slave-vkjp krb5kdc[20440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991421, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:41.212314 20839 init.cc:377] Logged in from keytab as kudu/127.31.166.131@KRBTEST.COM (short username kudu)
I20260806 04:43:41.213227 20839 webserver.cc:460] Webserver started at http://127.31.166.131:33207/ using document root <none> and password file <none>
I20260806 04:43:41.213850 20839 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:41.213958 20839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:41.214210 20839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:41.216558 20839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "bfde692fe3484b71bea5eb9a1250ed12"
format_stamp: "Formatted at 2026-08-06 04:43:41 on dist-test-slave-vkjp"
server_key: "db511908d8e79a0f307c0a824a7b3831"
server_key_iv: "91c75899e336d397d716b084356ef290"
server_key_version: "encryptionkey@0"
I20260806 04:43:41.217192 20839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "bfde692fe3484b71bea5eb9a1250ed12"
format_stamp: "Formatted at 2026-08-06 04:43:41 on dist-test-slave-vkjp"
server_key: "db511908d8e79a0f307c0a824a7b3831"
server_key_iv: "91c75899e336d397d716b084356ef290"
server_key_version: "encryptionkey@0"
I20260806 04:43:41.221195 20839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:43:41.223794 20855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:41.225075 20839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:43:41.225311 20839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "bfde692fe3484b71bea5eb9a1250ed12"
format_stamp: "Formatted at 2026-08-06 04:43:41 on dist-test-slave-vkjp"
server_key: "db511908d8e79a0f307c0a824a7b3831"
server_key_iv: "91c75899e336d397d716b084356ef290"
server_key_version: "encryptionkey@0"
I20260806 04:43:41.225430 20839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:41.268358 20839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:41.271543 20839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:41.271756 20839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:41.272382 20839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:41.273283 20839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:41.273355 20839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:41.273427 20839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:41.273480 20839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:41.282407 20839 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:45189
I20260806 04:43:41.282486 20978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:45189 every 8 connection(s)
I20260806 04:43:41.283355 20839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:43:41.287431 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 20839
I20260806 04:43:41.287609 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:43:41.287910 32410 external_mini_cluster.cc:1428] Setting key f17b3322f2cdb0251a5620a86051121b
Aug 06 04:43:41 dist-test-slave-vkjp krb5kdc[20440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991421, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.131@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:43:41.297107 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.285126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:57961 (local address 127.31.166.190:36141)
0806 04:43:41.285268 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:41.285272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:41.286074 (+ 802us) server_negotiation.cc:407] Connection header received
0806 04:43:41.287383 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:41.287386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:41.287479 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:41.287578 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:41.289137 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.289743 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.290977 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.291248 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.291333 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.293830 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:41.293833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:41.293835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:41.293884 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:41.295321 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:41.295794 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:41.295796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:41.295798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:41.295869 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:41.296168 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:41.296170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:41.296171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:41.296356 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:41.296434 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:41.296880 (+ 446us) server_negotiation.cc:299] Negotiation successful
0806 04:43:41.296981 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:43:41.298038 20981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.285476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:36141 (local address 127.31.166.131:57961)
0806 04:43:41.285924 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:41.285966 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:41.287155 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:41.287864 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:41.287870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:41.288371 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:41.288968 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.288983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.289929 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.289933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.290858 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.290869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.291405 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.291409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.291543 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.292155 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:43:41.292171 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:41.293647 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:41.295425 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:41.295429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:41.295440 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:41.295690 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:41.295949 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:43:41.295952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:43:41.295953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:43:41.296084 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:43:41.296447 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:41.296451 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:41.296670 (+ 219us) client_negotiation.cc:769] Sending connection context
0806 04:43:41.296827 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 04:43:41.297102 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
I20260806 04:43:41.299230 20979 heartbeater.cc:347] Connected to a master server at 127.31.166.190:36141
I20260806 04:43:41.299489 20979 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:41.299978 20979 heartbeater.cc:511] Master 127.31.166.190:36141 requested a full tablet report, sending...
I20260806 04:43:41.301095 20488 ts_manager.cc:194] Registered new tserver with Master: bfde692fe3484b71bea5eb9a1250ed12 (127.31.166.131:45189)
I20260806 04:43:41.301803 20488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.166.131@KRBTEST.COM'} at 127.31.166.131:57961
I20260806 04:43:41.301954 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:43:41.311233 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.304184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.31.166.190:36141)
0806 04:43:41.304295 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:41.304298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:41.304514 (+ 216us) server_negotiation.cc:407] Connection header received
0806 04:43:41.304607 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:41.304608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:41.304642 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:41.304705 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:41.305363 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.305843 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.306656 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.306871 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.306948 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.308331 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:41.308334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:41.308336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:41.308356 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:41.309617 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:41.310021 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:41.310024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:41.310025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:41.310098 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:41.310330 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:41.310332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:41.310333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:41.310585 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:41.310718 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:41.310950 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 04:43:41.311042 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:43:41.314194 20488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:43:41.317216 20488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:43:41.331589 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.325763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:45189 (local address 127.0.0.1:47752)
0806 04:43:41.326148 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:41.326162 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:41.326240 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:41.327066 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:41.327069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:41.327108 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:41.327207 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.327213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.328348 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.328353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.330116 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.330127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.330946 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.330950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.331111 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.331162 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 04:43:41.331302 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 04:43:41.331425 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":265,"thread_start_us":88,"threads_started":1}
I20260806 04:43:41.331729 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.325510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53262 (local address 127.31.166.129:34653)
0806 04:43:41.325918 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:41.325936 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:41.326111 (+ 175us) server_negotiation.cc:407] Connection header received
0806 04:43:41.326213 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:41.326220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:41.326431 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:41.326984 (+ 553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:41.327154 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.328321 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.330228 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.331108 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.331263 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.331376 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:41.331492 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 04:43:41.331582 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":93,"threads_started":1}
I20260806 04:43:41.332072 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.326165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.31.166.131:45189)
0806 04:43:41.326612 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:41.326625 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:41.326645 (+ 20us) server_negotiation.cc:407] Connection header received
0806 04:43:41.326720 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:41.326726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:41.326909 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:41.327066 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:41.327333 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.328218 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.330255 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.330823 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.330959 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.331132 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:41.331865 (+ 733us) server_negotiation.cc:299] Negotiation successful
0806 04:43:41.331970 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":165,"threads_started":1}
I20260806 04:43:41.331594 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.325357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:34653 (local address 127.0.0.1:53262)
0806 04:43:41.325718 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:41.325771 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:41.325899 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:41.326607 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:41.326614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:41.326642 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:41.326788 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.326797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.328451 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.328453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.330116 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.330127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.331270 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.331275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.331364 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.331385 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 04:43:41.331451 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:43:41.331505 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":148,"thread_start_us":89,"threads_started":1}
I20260806 04:43:41.332563 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.326052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:45281 (local address 127.0.0.1:37664)
0806 04:43:41.327031 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:41.327044 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:41.327236 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:41.327774 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:41.327778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:41.327796 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:41.327891 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.327896 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.329253 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.329256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.330816 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.330828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.332130 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.332134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.332240 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.332261 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 04:43:41.332329 (+ 68us) client_negotiation.cc:240] Negotiation successful
0806 04:43:41.332412 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":143,"thread_start_us":91,"threads_started":1}
I20260806 04:43:41.333010 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.326463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.31.166.130:45281)
0806 04:43:41.326862 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:41.326873 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:41.327160 (+ 287us) server_negotiation.cc:407] Connection header received
0806 04:43:41.327454 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:41.327461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:41.327626 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:41.327763 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:41.328003 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.328908 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.330965 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.331810 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.331917 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.332025 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:41.332770 (+ 745us) server_negotiation.cc:299] Negotiation successful
0806 04:43:41.332868 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":106,"threads_started":1}
I20260806 04:43:41.335355 20766 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9200077eb440e8ab58716ea1b02066 (DEFAULT_TABLE table=test-table [id=3279e012174a429593b9405a75d65820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:41.335357 20908 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9200077eb440e8ab58716ea1b02066 (DEFAULT_TABLE table=test-table [id=3279e012174a429593b9405a75d65820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:41.335534 20622 tablet_service.cc:1467] Processing CreateTablet for tablet 9b9200077eb440e8ab58716ea1b02066 (DEFAULT_TABLE table=test-table [id=3279e012174a429593b9405a75d65820]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:41.336376 20622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9200077eb440e8ab58716ea1b02066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:41.336376 20908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9200077eb440e8ab58716ea1b02066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:41.336382 20766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b9200077eb440e8ab58716ea1b02066. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:41.342545 20997 tablet_bootstrap.cc:492] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693: Bootstrap starting.
I20260806 04:43:41.344054 20999 tablet_bootstrap.cc:492] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12: Bootstrap starting.
I20260806 04:43:41.344826 20997 tablet_bootstrap.cc:654] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:41.345825 20999 tablet_bootstrap.cc:654] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:41.345899 20997 log.cc:824] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:41.346519 20998 tablet_bootstrap.cc:492] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164: Bootstrap starting.
I20260806 04:43:41.347221 20999 log.cc:824] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:41.348448 20997 tablet_bootstrap.cc:492] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693: No bootstrap required, opened a new log
I20260806 04:43:41.348646 20997 ts_tablet_manager.cc:1397] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 04:43:41.348879 20998 tablet_bootstrap.cc:654] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:41.349032 20999 tablet_bootstrap.cc:492] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12: No bootstrap required, opened a new log
I20260806 04:43:41.349236 20999 ts_tablet_manager.cc:1397] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:43:41.349682 20998 log.cc:824] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:41.351711 20998 tablet_bootstrap.cc:492] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164: No bootstrap required, opened a new log
I20260806 04:43:41.351909 20998 ts_tablet_manager.cc:1397] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260806 04:43:41.352448 20999 raft_consensus.cc:348] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e99364e33d46b6acc1aecc66dfe164" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34653 } } peers { permanent_uuid: "d6a8e23564db4bcab3cacc55d57c6693" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45281 } } peers { permanent_uuid: "bfde692fe3484b71bea5eb9a1250ed12" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 45189 } }
I20260806 04:43:41.352679 20999 raft_consensus.cc:374] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:41.352731 20999 raft_consensus.cc:729] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfde692fe3484b71bea5eb9a1250ed12, State: Initialized, Role: FOLLOWER
I20260806 04:43:41.352556 20997 raft_consensus.cc:348] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e99364e33d46b6acc1aecc66dfe164" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34653 } } peers { permanent_uuid: "d6a8e23564db4bcab3cacc55d57c6693" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45281 } } peers { permanent_uuid: "bfde692fe3484b71bea5eb9a1250ed12" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 45189 } }
I20260806 04:43:41.352798 20997 raft_consensus.cc:374] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:41.352864 20997 raft_consensus.cc:729] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a8e23564db4bcab3cacc55d57c6693, State: Initialized, Role: FOLLOWER
I20260806 04:43:41.353205 20999 consensus_queue.cc:260] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e99364e33d46b6acc1aecc66dfe164" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34653 } } peers { permanent_uuid: "d6a8e23564db4bcab3cacc55d57c6693" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45281 } } peers { permanent_uuid: "bfde692fe3484b71bea5eb9a1250ed12" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 45189 } }
I20260806 04:43:41.353274 20997 consensus_queue.cc:260] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e99364e33d46b6acc1aecc66dfe164" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34653 } } peers { permanent_uuid: "d6a8e23564db4bcab3cacc55d57c6693" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45281 } } peers { permanent_uuid: "bfde692fe3484b71bea5eb9a1250ed12" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 45189 } }
I20260806 04:43:41.354277 20999 ts_tablet_manager.cc:1428] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260806 04:43:41.354460 20997 ts_tablet_manager.cc:1428] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260806 04:43:41.354646 20979 heartbeater.cc:503] Master 127.31.166.190:36141 was elected leader, sending a full tablet report...
I20260806 04:43:41.354806 20834 heartbeater.cc:503] Master 127.31.166.190:36141 was elected leader, sending a full tablet report...
I20260806 04:43:41.356427 20998 raft_consensus.cc:348] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e99364e33d46b6acc1aecc66dfe164" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34653 } } peers { permanent_uuid: "d6a8e23564db4bcab3cacc55d57c6693" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45281 } } peers { permanent_uuid: "bfde692fe3484b71bea5eb9a1250ed12" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 45189 } }
I20260806 04:43:41.356698 20998 raft_consensus.cc:374] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:41.356763 20998 raft_consensus.cc:729] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e99364e33d46b6acc1aecc66dfe164, State: Initialized, Role: FOLLOWER
I20260806 04:43:41.357225 20998 consensus_queue.cc:260] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e99364e33d46b6acc1aecc66dfe164" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34653 } } peers { permanent_uuid: "d6a8e23564db4bcab3cacc55d57c6693" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45281 } } peers { permanent_uuid: "bfde692fe3484b71bea5eb9a1250ed12" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 45189 } }
I20260806 04:43:41.358088 20694 heartbeater.cc:503] Master 127.31.166.190:36141 was elected leader, sending a full tablet report...
I20260806 04:43:41.358243 20998 ts_tablet_manager.cc:1428] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260806 04:43:41.409821 21004 raft_consensus.cc:482] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:43:41.410128 21004 raft_consensus.cc:504] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e99364e33d46b6acc1aecc66dfe164" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34653 } } peers { permanent_uuid: "d6a8e23564db4bcab3cacc55d57c6693" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45281 } } peers { permanent_uuid: "bfde692fe3484b71bea5eb9a1250ed12" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 45189 } }
I20260806 04:43:41.411643 21004 leader_election.cc:288] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60e99364e33d46b6acc1aecc66dfe164 (127.31.166.129:34653), bfde692fe3484b71bea5eb9a1250ed12 (127.31.166.131:45189)
I20260806 04:43:41.416499 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.412200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:56061 (local address 127.31.166.131:45189)
0806 04:43:41.412367 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:41.412371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:41.412450 (+ 79us) server_negotiation.cc:407] Connection header received
0806 04:43:41.412605 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:41.412609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:41.412669 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:41.412775 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:41.413087 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.413847 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.415260 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.415810 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.415946 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.416008 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:41.416248 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 04:43:41.416355 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 04:43:41.416565 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.411915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:45189 (local address 127.31.166.130:56061)
0806 04:43:41.412338 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:41.412365 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:41.412467 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:41.412796 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:41.412800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:41.412821 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:41.412962 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.412969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.414020 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.414024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.415122 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.415135 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.415954 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.415959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.416084 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.416121 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 04:43:41.416205 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 04:43:41.416293 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":28,"thread_start_us":148,"threads_started":1}
I20260806 04:43:41.416996 20836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.411837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:34653 (local address 127.31.166.130:46193)
0806 04:43:41.412051 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:41.412069 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:41.412176 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:41.412640 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:41.412644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:41.412664 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:41.412823 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.412832 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.413971 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.413976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.415440 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.415452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.416507 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.416514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.416632 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.416666 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 04:43:41.416777 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 04:43:41.416859 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":53}
I20260806 04:43:41.416996 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.412226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:46193 (local address 127.31.166.129:34653)
0806 04:43:41.412372 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:41.412378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:41.412395 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:43:41.412458 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:41.412461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:41.412518 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:41.412595 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:41.412996 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.413849 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.415608 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.416236 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.416376 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.416440 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:41.416798 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 04:43:41.416878 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:43:41.417250 20929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9200077eb440e8ab58716ea1b02066" candidate_uuid: "d6a8e23564db4bcab3cacc55d57c6693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfde692fe3484b71bea5eb9a1250ed12" is_pre_election: true
I20260806 04:43:41.417608 20929 raft_consensus.cc:2454] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6a8e23564db4bcab3cacc55d57c6693 in term 0.
I20260806 04:43:41.417735 20646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9200077eb440e8ab58716ea1b02066" candidate_uuid: "d6a8e23564db4bcab3cacc55d57c6693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60e99364e33d46b6acc1aecc66dfe164" is_pre_election: true
I20260806 04:43:41.418035 20646 raft_consensus.cc:2454] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6a8e23564db4bcab3cacc55d57c6693 in term 0.
I20260806 04:43:41.418128 20718 leader_election.cc:302] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfde692fe3484b71bea5eb9a1250ed12, d6a8e23564db4bcab3cacc55d57c6693; no voters:
I20260806 04:43:41.418458 21004 raft_consensus.cc:2781] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:43:41.418560 21004 raft_consensus.cc:482] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:43:41.418604 21004 raft_consensus.cc:3037] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:41.419767 21004 raft_consensus.cc:504] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e99364e33d46b6acc1aecc66dfe164" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34653 } } peers { permanent_uuid: "d6a8e23564db4bcab3cacc55d57c6693" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45281 } } peers { permanent_uuid: "bfde692fe3484b71bea5eb9a1250ed12" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 45189 } }
I20260806 04:43:41.420169 21004 leader_election.cc:288] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [CANDIDATE]: Term 1 election: Requested vote from peers 60e99364e33d46b6acc1aecc66dfe164 (127.31.166.129:34653), bfde692fe3484b71bea5eb9a1250ed12 (127.31.166.131:45189)
I20260806 04:43:41.420513 20929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9200077eb440e8ab58716ea1b02066" candidate_uuid: "d6a8e23564db4bcab3cacc55d57c6693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfde692fe3484b71bea5eb9a1250ed12"
I20260806 04:43:41.420527 20646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b9200077eb440e8ab58716ea1b02066" candidate_uuid: "d6a8e23564db4bcab3cacc55d57c6693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60e99364e33d46b6acc1aecc66dfe164"
I20260806 04:43:41.420660 20929 raft_consensus.cc:3037] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:41.420666 20646 raft_consensus.cc:3037] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:41.421854 20929 raft_consensus.cc:2454] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6a8e23564db4bcab3cacc55d57c6693 in term 1.
I20260806 04:43:41.422181 20718 leader_election.cc:302] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfde692fe3484b71bea5eb9a1250ed12, d6a8e23564db4bcab3cacc55d57c6693; no voters:
I20260806 04:43:41.422242 20646 raft_consensus.cc:2454] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6a8e23564db4bcab3cacc55d57c6693 in term 1.
I20260806 04:43:41.422377 21004 raft_consensus.cc:2781] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:41.422617 21004 raft_consensus.cc:686] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [term 1 LEADER]: Becoming Leader. State: Replica: d6a8e23564db4bcab3cacc55d57c6693, State: Running, Role: LEADER
I20260806 04:43:41.423009 21004 consensus_queue.cc:237] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e99364e33d46b6acc1aecc66dfe164" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34653 } } peers { permanent_uuid: "d6a8e23564db4bcab3cacc55d57c6693" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45281 } } peers { permanent_uuid: "bfde692fe3484b71bea5eb9a1250ed12" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 45189 } }
I20260806 04:43:41.426457 20487 catalog_manager.cc:5495] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6a8e23564db4bcab3cacc55d57c6693 (127.31.166.130). New cstate: current_term: 1 leader_uuid: "d6a8e23564db4bcab3cacc55d57c6693" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e99364e33d46b6acc1aecc66dfe164" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6a8e23564db4bcab3cacc55d57c6693" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 45281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bfde692fe3484b71bea5eb9a1250ed12" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 45189 } health_report { overall_health: UNKNOWN } } }
I20260806 04:43:41.438999 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.434844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37672 (local address 127.31.166.130:45281)
0806 04:43:41.434984 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:41.434991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:41.435010 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:43:41.435124 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:41.435129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:41.435216 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:41.435353 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:43:41.435535 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.436374 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.437208 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.437617 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.437756 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.437978 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:43:41.438536 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:43:41.438676 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:41.438729 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 04:43:41.438812 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 04:43:41.445645 20646 raft_consensus.cc:1261] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164 [term 1 FOLLOWER]: Refusing update from remote peer d6a8e23564db4bcab3cacc55d57c6693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:43:41.445706 20929 raft_consensus.cc:1261] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12 [term 1 FOLLOWER]: Refusing update from remote peer d6a8e23564db4bcab3cacc55d57c6693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:43:41.446455 21007 consensus_queue.cc:1035] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e99364e33d46b6acc1aecc66dfe164" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:43:41.446696 21004 consensus_queue.cc:1035] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfde692fe3484b71bea5eb9a1250ed12" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:43:41.457173 21012 mvcc.cc:205] Tried to move back new op lower bound from 7315420862233931776 to 7315420862151737344. Current Snapshot: MvccSnapshot[applied={T|T < 7315420862233931776 or (T in {7315420862233931776})}]
I20260806 04:43:41.465363 20488 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:43:41.465585 20488 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:43:41.468787 20488 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f68c1aebf5f8406a81795eddac5f76b9: Sending DeleteTablet for 3 replicas of tablet 9b9200077eb440e8ab58716ea1b02066
I20260806 04:43:41.469792 20622 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b9200077eb440e8ab58716ea1b02066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:41 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:53262
I20260806 04:43:41.469897 20908 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b9200077eb440e8ab58716ea1b02066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:41 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:47752
I20260806 04:43:41.470144 20766 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b9200077eb440e8ab58716ea1b02066 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:41 UTC) from {username='kudu', principal='kudu/127.31.166.190@KRBTEST.COM'} at 127.0.0.1:37664
I20260806 04:43:41.470474 21021 tablet_replica.cc:330] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164: stopping tablet replica
I20260806 04:43:41.470692 21022 tablet_replica.cc:330] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693: stopping tablet replica
I20260806 04:43:41.471055 21021 raft_consensus.cc:2229] T 9b9200077eb440e8ab58716ea1b02066 P 60e99364e33d46b6acc1aecc66dfe164 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:43:41.471051 21022 raft_consensus.cc:2229] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:43:41.471241 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 20556
I20260806 04:43:41.471549 21022 raft_consensus.cc:2258] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:43:41.474387 21022 ts_tablet_manager.cc:1905] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:43:41.474431 21020 tablet_replica.cc:330] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12: stopping tablet replica
I20260806 04:43:41.474887 21020 raft_consensus.cc:2229] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:43:41.475275 21020 raft_consensus.cc:2258] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:43:41.476703 21020 ts_tablet_manager.cc:1905] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:43:41.478175 21022 ts_tablet_manager.cc:1918] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:43:41.478305 21022 log.cc:1196] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/9b9200077eb440e8ab58716ea1b02066
I20260806 04:43:41.478700 21022 ts_tablet_manager.cc:1939] T 9b9200077eb440e8ab58716ea1b02066 P d6a8e23564db4bcab3cacc55d57c6693: Deleting consensus metadata
I20260806 04:43:41.479647 21020 ts_tablet_manager.cc:1918] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:43:41.479800 21020 log.cc:1196] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/9b9200077eb440e8ab58716ea1b02066
I20260806 04:43:41.480093 20473 catalog_manager.cc:4841] TS d6a8e23564db4bcab3cacc55d57c6693 (127.31.166.130:45281): tablet 9b9200077eb440e8ab58716ea1b02066 (table test-table [id=3279e012174a429593b9405a75d65820]) successfully deleted
I20260806 04:43:41.480264 21020 ts_tablet_manager.cc:1939] T 9b9200077eb440e8ab58716ea1b02066 P bfde692fe3484b71bea5eb9a1250ed12: Deleting consensus metadata
I20260806 04:43:41.481436 20472 catalog_manager.cc:4841] TS bfde692fe3484b71bea5eb9a1250ed12 (127.31.166.131:45189): tablet 9b9200077eb440e8ab58716ea1b02066 (table test-table [id=3279e012174a429593b9405a75d65820]) successfully deleted
W20260806 04:43:41.483263 20473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:34653 (error 108)
I20260806 04:43:41.483551 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 20699
I20260806 04:43:41.484972 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.484650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:34653 (local address 127.0.0.1:53270)
0806 04:43:41.484812 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:41.484887 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:34653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260806 04:43:41.485216 20473 catalog_manager.cc:4584] TS 60e99364e33d46b6acc1aecc66dfe164 (127.31.166.129:34653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b9200077eb440e8ab58716ea1b02066: Network error: Client connection negotiation failed: client connection to 127.31.166.129:34653: connect: Connection refused (error 111)
I20260806 04:43:41.490087 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 20839
I20260806 04:43:41.495802 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 20454
2026-08-06T04:43:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2362 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T04:43:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:41Z Disabled control of system clock
I20260806 04:43:41.527891 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:39599
--webserver_interface=127.31.166.190
--webserver_port=0
--builtin_ntp_servers=127.31.166.148:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:39599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:43:41.641397 21027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:41.641669 21027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:41.641705 21027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:41.645753 21027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:41.645812 21027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:41.645828 21027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:41.645845 21027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:41.645860 21027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:41.650449 21027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:39599
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:39599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:41.651484 21027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:41.652837 21027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:41.661219 21033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:41.661213 21032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:41.661356 21027 server_base.cc:1034] running on GCE node
W20260806 04:43:41.661213 21035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:41.662048 21027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:41.663232 21027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:41.664417 21027 hybrid_clock.cc:648] HybridClock initialized: now 1785991421664398 us; error 29 us; skew 500 ppm
I20260806 04:43:41.666251 21027 webserver.cc:460] Webserver started at http://127.31.166.190:35197/ using document root <none> and password file <none>
I20260806 04:43:41.666831 21027 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:41.666896 21027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:41.667181 21027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:41.669167 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "ad2afaa1d1e14ef1b345ee7854f7a845"
format_stamp: "Formatted at 2026-08-06 04:43:41 on dist-test-slave-vkjp"
server_key: "143a6a968f52567be7f12eb44e5136fb"
server_key_iv: "0fd1ac20da44348dda737fe54ff30b00"
server_key_version: "encryptionkey@0"
I20260806 04:43:41.669775 21027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "ad2afaa1d1e14ef1b345ee7854f7a845"
format_stamp: "Formatted at 2026-08-06 04:43:41 on dist-test-slave-vkjp"
server_key: "143a6a968f52567be7f12eb44e5136fb"
server_key_iv: "0fd1ac20da44348dda737fe54ff30b00"
server_key_version: "encryptionkey@0"
I20260806 04:43:41.673689 21027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 04:43:41.676129 21042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:41.677341 21027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:43:41.677487 21027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "ad2afaa1d1e14ef1b345ee7854f7a845"
format_stamp: "Formatted at 2026-08-06 04:43:41 on dist-test-slave-vkjp"
server_key: "143a6a968f52567be7f12eb44e5136fb"
server_key_iv: "0fd1ac20da44348dda737fe54ff30b00"
server_key_version: "encryptionkey@0"
I20260806 04:43:41.677592 21027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:41.690131 21027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:41.690830 21027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:41.691042 21027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:41.698351 21027 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:39599
I20260806 04:43:41.698376 21096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:39599 every 8 connection(s)
I20260806 04:43:41.699400 21027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:41.702610 21097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:41.703744 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 21027
I20260806 04:43:41.703890 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:41.704196 32410 external_mini_cluster.cc:1428] Setting key 3e1040bca5787c51cddb049e647b1cd1
I20260806 04:43:41.709117 21097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845: Bootstrap starting.
I20260806 04:43:41.712023 21097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:41.712786 21097 log.cc:824] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:41.714105 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.705656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56106 (local address 127.31.166.190:39599)
0806 04:43:41.706254 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:41.706269 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:41.706303 (+ 34us) server_negotiation.cc:407] Connection header received
0806 04:43:41.706925 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:41.706942 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:41.707393 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:41.707658 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:41.708069 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.709458 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.710750 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.711276 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.711431 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.712272 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:41.712277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:41.712289 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:41.712682 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:41.712737 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:41.712758 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:41.712837 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:41.713043 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 04:43:41.713293 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":141,"threads_started":1}
I20260806 04:43:41.714870 21097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845: No bootstrap required, opened a new log
I20260806 04:43:41.718541 21097 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2afaa1d1e14ef1b345ee7854f7a845" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39599 } }
I20260806 04:43:41.718837 21097 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:41.718910 21097 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2afaa1d1e14ef1b345ee7854f7a845, State: Initialized, Role: FOLLOWER
I20260806 04:43:41.719431 21097 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2afaa1d1e14ef1b345ee7854f7a845" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39599 } }
I20260806 04:43:41.719554 21097 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:41.719638 21097 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:41.719753 21097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:41.720726 21097 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2afaa1d1e14ef1b345ee7854f7a845" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39599 } }
I20260806 04:43:41.721081 21097 leader_election.cc:302] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad2afaa1d1e14ef1b345ee7854f7a845; no voters:
I20260806 04:43:41.721412 21097 leader_election.cc:288] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:41.721581 21102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:41.721859 21102 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [term 1 LEADER]: Becoming Leader. State: Replica: ad2afaa1d1e14ef1b345ee7854f7a845, State: Running, Role: LEADER
I20260806 04:43:41.722198 21102 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2afaa1d1e14ef1b345ee7854f7a845" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39599 } }
I20260806 04:43:41.722755 21097 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:41.724141 21103 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad2afaa1d1e14ef1b345ee7854f7a845" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2afaa1d1e14ef1b345ee7854f7a845" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39599 } } }
I20260806 04:43:41.724148 21105 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad2afaa1d1e14ef1b345ee7854f7a845. Latest consensus state: current_term: 1 leader_uuid: "ad2afaa1d1e14ef1b345ee7854f7a845" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2afaa1d1e14ef1b345ee7854f7a845" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39599 } } }
I20260806 04:43:41.724301 21103 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:41.724301 21105 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:41.724822 21115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:41.727519 21115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:41.732859 21115 catalog_manager.cc:1347] Generated new cluster ID: 58c668b16f73411d9a4478b68fbfbf6c
I20260806 04:43:41.732928 21115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:41.743359 21115 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:41.744297 21115 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:41.750739 21115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845: Generated new TSK 0
I20260806 04:43:41.751479 21115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:43:41.756662 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.129:0
--local_ip_for_outbound_sockets=127.31.166.129
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39599
--builtin_ntp_servers=127.31.166.148:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 04:43:41.867843 21123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:41.868086 21123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:41.868125 21123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:41.872324 21123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:41.872376 21123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:41.872438 21123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.129
I20260806 04:43:41.877075 21123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.166.129
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.129
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:41.878031 21123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:41.879331 21123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:41.888911 21128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:41.888933 21129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:41.888967 21131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:41.889074 21123 server_base.cc:1034] running on GCE node
I20260806 04:43:41.889616 21123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:41.890213 21123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:41.891376 21123 hybrid_clock.cc:648] HybridClock initialized: now 1785991421891362 us; error 24 us; skew 500 ppm
I20260806 04:43:41.893230 21123 webserver.cc:460] Webserver started at http://127.31.166.129:37557/ using document root <none> and password file <none>
I20260806 04:43:41.893793 21123 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:41.893857 21123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:41.894084 21123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:41.896178 21123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/data/instance:
uuid: "eb9501d3b9d2405b9cafec880f182961"
format_stamp: "Formatted at 2026-08-06 04:43:41 on dist-test-slave-vkjp"
server_key: "e886f90978dbeae6fe61bec4d1960c90"
server_key_iv: "a1448859cd0d0ed3b892de80c0ec09d6"
server_key_version: "encryptionkey@0"
I20260806 04:43:41.896740 21123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance:
uuid: "eb9501d3b9d2405b9cafec880f182961"
format_stamp: "Formatted at 2026-08-06 04:43:41 on dist-test-slave-vkjp"
server_key: "e886f90978dbeae6fe61bec4d1960c90"
server_key_iv: "a1448859cd0d0ed3b892de80c0ec09d6"
server_key_version: "encryptionkey@0"
I20260806 04:43:41.900512 21123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:43:41.903020 21137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:41.904263 21123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:43:41.904407 21123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/wal
uuid: "eb9501d3b9d2405b9cafec880f182961"
format_stamp: "Formatted at 2026-08-06 04:43:41 on dist-test-slave-vkjp"
server_key: "e886f90978dbeae6fe61bec4d1960c90"
server_key_iv: "a1448859cd0d0ed3b892de80c0ec09d6"
server_key_version: "encryptionkey@0"
I20260806 04:43:41.904520 21123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:41.920470 21123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:41.921226 21123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:41.921440 21123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:41.922053 21123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:41.923120 21123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:41.923193 21123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:41.923273 21123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:41.923305 21123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:41.932262 21123 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.129:39505
I20260806 04:43:41.932288 21256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.129:39505 every 8 connection(s)
I20260806 04:43:41.933185 21123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/data/info.pb
I20260806 04:43:41.943159 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.935035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:56799 (local address 127.31.166.190:39599)
0806 04:43:41.935221 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:41.935225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:41.936259 (+ 1034us) server_negotiation.cc:407] Connection header received
0806 04:43:41.937680 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:41.937684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:41.937744 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:41.937854 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:41.939243 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.939924 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.941113 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.941352 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.941467 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.942607 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:41.942611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:41.942614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:41.942627 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:41.942656 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:41.942665 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:41.942744 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:41.942910 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 04:43:41.942984 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260806 04:43:41.943349 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 21123
I20260806 04:43:41.943492 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-0/wal/instance
I20260806 04:43:41.943828 32410 external_mini_cluster.cc:1428] Setting key c2acd32352f1c0ccd44b94eefbbc26ba
I20260806 04:43:41.944149 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:41.935461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39599 (local address 127.31.166.129:56799)
0806 04:43:41.936106 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:41.936147 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:41.937467 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:41.938066 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:41.938076 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:41.938202 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:41.939046 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.939066 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.940062 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.940065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.940972 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:41.940981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:41.941514 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:41.941518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:41.941696 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:41.942340 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:43:41.942341 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:41.942384 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:43:41.942447 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:41.942747 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:41.942759 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:41.942766 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:43:41.942910 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 04:43:41.943215 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260806 04:43:41.945467 21257 heartbeater.cc:347] Connected to a master server at 127.31.166.190:39599
I20260806 04:43:41.945840 21257 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:41.946029 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.130:0
--local_ip_for_outbound_sockets=127.31.166.130
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39599
--builtin_ntp_servers=127.31.166.148:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:43:41.946461 21257 heartbeater.cc:511] Master 127.31.166.190:39599 requested a full tablet report, sending...
I20260806 04:43:41.948057 21060 ts_manager.cc:194] Registered new tserver with Master: eb9501d3b9d2405b9cafec880f182961 (127.31.166.129:39505)
I20260806 04:43:41.949194 21060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.166.129:56799
W20260806 04:43:42.057463 21261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:42.057790 21261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:42.057862 21261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:42.061849 21261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:42.061933 21261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:42.062026 21261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.130
I20260806 04:43:42.066980 21261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.166.130
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.130
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:42.068073 21261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:42.069442 21261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:42.078706 21266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:42.078718 21267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:42.078718 21269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:42.079015 21261 server_base.cc:1034] running on GCE node
I20260806 04:43:42.079478 21261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:42.080130 21261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:42.081290 21261 hybrid_clock.cc:648] HybridClock initialized: now 1785991422081266 us; error 44 us; skew 500 ppm
I20260806 04:43:42.083300 21261 webserver.cc:460] Webserver started at http://127.31.166.130:33071/ using document root <none> and password file <none>
I20260806 04:43:42.083865 21261 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:42.083954 21261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:42.084183 21261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:42.086172 21261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/data/instance:
uuid: "f1820e7a19bc49d6a1a4aace9570373c"
format_stamp: "Formatted at 2026-08-06 04:43:42 on dist-test-slave-vkjp"
server_key: "34caef411e2a4542909ddfb9cbaa34e0"
server_key_iv: "01fad3696777cac75a5d1455baac993b"
server_key_version: "encryptionkey@0"
I20260806 04:43:42.086740 21261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance:
uuid: "f1820e7a19bc49d6a1a4aace9570373c"
format_stamp: "Formatted at 2026-08-06 04:43:42 on dist-test-slave-vkjp"
server_key: "34caef411e2a4542909ddfb9cbaa34e0"
server_key_iv: "01fad3696777cac75a5d1455baac993b"
server_key_version: "encryptionkey@0"
I20260806 04:43:42.090317 21261 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 04:43:42.092878 21275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:42.094151 21261 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:43:42.094283 21261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/wal
uuid: "f1820e7a19bc49d6a1a4aace9570373c"
format_stamp: "Formatted at 2026-08-06 04:43:42 on dist-test-slave-vkjp"
server_key: "34caef411e2a4542909ddfb9cbaa34e0"
server_key_iv: "01fad3696777cac75a5d1455baac993b"
server_key_version: "encryptionkey@0"
I20260806 04:43:42.094417 21261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:42.131325 21261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:42.132079 21261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:42.132328 21261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:42.132983 21261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:42.133951 21261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:42.134032 21261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:42.134105 21261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:42.134150 21261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:42.143813 21261 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.130:39775
I20260806 04:43:42.143841 21393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.130:39775 every 8 connection(s)
I20260806 04:43:42.144708 21261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/data/info.pb
I20260806 04:43:42.152897 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 21261
I20260806 04:43:42.153057 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/wal/instance
I20260806 04:43:42.153380 32410 external_mini_cluster.cc:1428] Setting key 1ee0c56b34006f68bab7f593e1801eca
I20260806 04:43:42.154157 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.146591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.130:56253 (local address 127.31.166.190:39599)
0806 04:43:42.146740 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:42.146744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:42.147769 (+ 1025us) server_negotiation.cc:407] Connection header received
0806 04:43:42.149218 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:42.149221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:42.149286 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:42.149399 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:42.150302 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.150831 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.152050 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.152283 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.152411 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.153627 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:42.153630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:42.153632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:42.153646 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:42.153667 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:42.153676 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:42.153755 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:42.153937 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 04:43:42.154011 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:43:42.155027 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.146852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39599 (local address 127.31.166.130:56253)
0806 04:43:42.147596 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:42.147639 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:42.148983 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:42.149573 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:42.149581 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:42.149664 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:42.150142 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.150154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.150965 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.150969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.151917 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.151928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.152449 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.152453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.152626 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.153297 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:43:42.153299 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:42.153352 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:43:42.153441 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:42.153757 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:42.153770 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:42.153777 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 04:43:42.153937 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 04:43:42.154223 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":122,"threads_started":1}
I20260806 04:43:42.155566 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.166.131:0
--local_ip_for_outbound_sockets=127.31.166.131
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39599
--builtin_ntp_servers=127.31.166.148:36331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 04:43:42.156229 21394 heartbeater.cc:347] Connected to a master server at 127.31.166.190:39599
I20260806 04:43:42.156479 21394 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:42.157111 21394 heartbeater.cc:511] Master 127.31.166.190:39599 requested a full tablet report, sending...
I20260806 04:43:42.158267 21060 ts_manager.cc:194] Registered new tserver with Master: f1820e7a19bc49d6a1a4aace9570373c (127.31.166.130:39775)
I20260806 04:43:42.159150 21060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.166.130:56253
W20260806 04:43:42.266986 21397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:42.267253 21397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:42.267292 21397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:42.271152 21397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:42.271198 21397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:42.271248 21397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.166.131
I20260806 04:43:42.276134 21397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:36331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.166.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.166.131
--webserver_port=0
--tserver_master_addrs=127.31.166.190:39599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.166.131
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:42.277132 21397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:42.278430 21397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:42.287933 21403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:42.287940 21405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:42.287933 21402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:42.288311 21397 server_base.cc:1034] running on GCE node
I20260806 04:43:42.288722 21397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:42.289371 21397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:42.290563 21397 hybrid_clock.cc:648] HybridClock initialized: now 1785991422290553 us; error 30 us; skew 500 ppm
I20260806 04:43:42.292543 21397 webserver.cc:460] Webserver started at http://127.31.166.131:36375/ using document root <none> and password file <none>
I20260806 04:43:42.293120 21397 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:42.293213 21397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:42.293449 21397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:42.295548 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/data/instance:
uuid: "73aad0d4724a444283b5ef30dac489a2"
format_stamp: "Formatted at 2026-08-06 04:43:42 on dist-test-slave-vkjp"
server_key: "64d1f4be2785771b0ad0485733f6373e"
server_key_iv: "ab03afbece80d438cb36b2498878f4f1"
server_key_version: "encryptionkey@0"
I20260806 04:43:42.296118 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance:
uuid: "73aad0d4724a444283b5ef30dac489a2"
format_stamp: "Formatted at 2026-08-06 04:43:42 on dist-test-slave-vkjp"
server_key: "64d1f4be2785771b0ad0485733f6373e"
server_key_iv: "ab03afbece80d438cb36b2498878f4f1"
server_key_version: "encryptionkey@0"
I20260806 04:43:42.299835 21397 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:43:42.302182 21413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:42.303463 21397 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:43:42.303565 21397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/wal
uuid: "73aad0d4724a444283b5ef30dac489a2"
format_stamp: "Formatted at 2026-08-06 04:43:42 on dist-test-slave-vkjp"
server_key: "64d1f4be2785771b0ad0485733f6373e"
server_key_iv: "ab03afbece80d438cb36b2498878f4f1"
server_key_version: "encryptionkey@0"
I20260806 04:43:42.303646 21397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:42.314917 21397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:42.315570 21397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:42.315719 21397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:42.316211 21397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 04:43:42.317073 21397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 04:43:42.317121 21397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:42.317159 21397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 04:43:42.317175 21397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:42.326120 21397 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.131:35419
I20260806 04:43:42.326179 21531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.131:35419 every 8 connection(s)
I20260806 04:43:42.327044 21397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/data/info.pb
I20260806 04:43:42.332835 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 21397
I20260806 04:43:42.332947 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/wal/instance
I20260806 04:43:42.333304 32410 external_mini_cluster.cc:1428] Setting key 4efbde940daf5d3120fa627d19dc1d14
I20260806 04:43:42.336867 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.328836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:52529 (local address 127.31.166.190:39599)
0806 04:43:42.329002 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:42.329007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:42.330051 (+ 1044us) server_negotiation.cc:407] Connection header received
0806 04:43:42.331446 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:42.331450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:42.331502 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:42.331593 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:42.332438 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.333117 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.334736 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.335117 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.335242 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.336438 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:42.336442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:42.336444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:42.336456 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:42.336477 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:42.336486 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:42.336551 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:42.336687 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 04:43:42.336739 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 04:43:42.337668 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.329236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39599 (local address 127.31.166.131:52529)
0806 04:43:42.329909 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:42.329951 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:42.331246 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:42.331742 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:42.331748 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:42.331826 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:43:42.332304 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.332317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.333248 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.333252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.334597 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.334608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.335284 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.335291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.335546 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.336181 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 04:43:42.336183 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:43:42.336221 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 04:43:42.336279 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:43:42.336563 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:43:42.336570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:43:42.336574 (+ 4us) client_negotiation.cc:769] Sending connection context
0806 04:43:42.336664 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 04:43:42.336874 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":115,"threads_started":1}
I20260806 04:43:42.338830 21532 heartbeater.cc:347] Connected to a master server at 127.31.166.190:39599
I20260806 04:43:42.339171 21532 heartbeater.cc:464] Registering TS with master...
I20260806 04:43:42.339730 21532 heartbeater.cc:511] Master 127.31.166.190:39599 requested a full tablet report, sending...
I20260806 04:43:42.340837 21060 ts_manager.cc:194] Registered new tserver with Master: 73aad0d4724a444283b5ef30dac489a2 (127.31.166.131:35419)
I20260806 04:43:42.341679 21060 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.166.131:52529
I20260806 04:43:42.347389 32410 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 04:43:42.353152 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.349482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.31.166.190:39599)
0806 04:43:42.349625 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:42.349628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:42.349814 (+ 186us) server_negotiation.cc:407] Connection header received
0806 04:43:42.349899 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:42.349901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:42.349942 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:42.350015 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:42.350238 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.350772 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.351645 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.351896 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.351986 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.352801 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:42.352805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:42.352807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:42.352820 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:42.352839 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:42.352847 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:42.352912 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:42.352990 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 04:43:42.353041 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 04:43:42.356083 21060 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 04:43:42.359063 21060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 04:43:42.374329 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.368196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:35419 (local address 127.0.0.1:46370)
0806 04:43:42.369418 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:42.369435 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:42.369533 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:42.370105 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:42.370111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:42.370132 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:42.370266 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.370272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.371705 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.371708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.372799 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.372809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.373742 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.373746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.373863 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.373907 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 04:43:42.374023 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 04:43:42.374147 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1151,"mutex_wait_us":1,"spinlock_wait_cycles":9984,"thread_start_us":149,"threads_started":1}
I20260806 04:43:42.374342 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.369285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.31.166.131:35419)
0806 04:43:42.369614 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:42.369625 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:42.369650 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:43:42.369734 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:42.369742 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:42.369953 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:42.370112 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:42.370382 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.371536 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.372937 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.373618 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.373741 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.373918 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:42.374071 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 04:43:42.374177 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":150,"threads_started":1}
I20260806 04:43:42.374958 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.368923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:39505 (local address 127.0.0.1:53754)
0806 04:43:42.369688 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:42.369701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:42.369779 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:42.370372 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:42.370375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:42.370406 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:42.370620 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.370627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.372235 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.372238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.373329 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.373336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.374534 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.374538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.374621 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.374651 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 04:43:42.374727 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 04:43:42.374805 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":47,"spinlock_wait_cycles":7296,"thread_start_us":88,"threads_started":1}
I20260806 04:43:42.375197 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.368315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.31.166.130:39775)
0806 04:43:42.368766 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:42.368779 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:42.369655 (+ 876us) server_negotiation.cc:407] Connection header received
0806 04:43:42.369865 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:42.369871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:42.370068 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:42.370223 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:42.370783 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.371792 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.373266 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.374052 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.374167 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.374317 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:42.374908 (+ 591us) server_negotiation.cc:299] Negotiation successful
0806 04:43:42.375006 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":133,"threads_started":1}
I20260806 04:43:42.374791 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.368172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:39775 (local address 127.0.0.1:52132)
0806 04:43:42.369050 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:42.369100 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:42.369699 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:42.370372 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:42.370375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:42.370407 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:42.370623 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.370627 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.371926 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.371931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.373137 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.373147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.374391 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.374394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.374487 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.374507 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 04:43:42.374573 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 04:43:42.374636 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":69,"spinlock_wait_cycles":5376,"thread_start_us":76,"threads_started":1}
I20260806 04:43:42.375507 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.369329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53754 (local address 127.31.166.129:39505)
0806 04:43:42.369840 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:42.369852 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:42.369873 (+ 21us) server_negotiation.cc:407] Connection header received
0806 04:43:42.369937 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:42.369941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:42.370082 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:42.370223 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:42.370762 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.372113 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.373444 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.374165 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.374278 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.374456 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:42.375190 (+ 734us) server_negotiation.cc:299] Negotiation successful
0806 04:43:42.375290 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":128,"threads_started":1}
I20260806 04:43:42.377799 21187 tablet_service.cc:1467] Processing CreateTablet for tablet 90b0602a09b544b3a9a573e04a3fef81 (DEFAULT_TABLE table=test-table [id=b03dda6b966f42bb94504fd510915d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:42.377960 21323 tablet_service.cc:1467] Processing CreateTablet for tablet 90b0602a09b544b3a9a573e04a3fef81 (DEFAULT_TABLE table=test-table [id=b03dda6b966f42bb94504fd510915d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:42.377926 21462 tablet_service.cc:1467] Processing CreateTablet for tablet 90b0602a09b544b3a9a573e04a3fef81 (DEFAULT_TABLE table=test-table [id=b03dda6b966f42bb94504fd510915d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 04:43:42.378808 21187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b0602a09b544b3a9a573e04a3fef81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:42.378973 21323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b0602a09b544b3a9a573e04a3fef81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:42.378973 21462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b0602a09b544b3a9a573e04a3fef81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:42.384857 21549 tablet_bootstrap.cc:492] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961: Bootstrap starting.
I20260806 04:43:42.386503 21551 tablet_bootstrap.cc:492] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c: Bootstrap starting.
I20260806 04:43:42.387036 21552 tablet_bootstrap.cc:492] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2: Bootstrap starting.
I20260806 04:43:42.387236 21549 tablet_bootstrap.cc:654] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:42.388132 21549 log.cc:824] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:42.388581 21551 tablet_bootstrap.cc:654] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:42.389195 21552 tablet_bootstrap.cc:654] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:42.389389 21551 log.cc:824] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:42.389950 21552 log.cc:824] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:42.390885 21549 tablet_bootstrap.cc:492] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961: No bootstrap required, opened a new log
I20260806 04:43:42.391116 21549 ts_tablet_manager.cc:1397] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260806 04:43:42.391480 21551 tablet_bootstrap.cc:492] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c: No bootstrap required, opened a new log
I20260806 04:43:42.391672 21551 ts_tablet_manager.cc:1397] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:43:42.391902 21552 tablet_bootstrap.cc:492] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2: No bootstrap required, opened a new log
I20260806 04:43:42.392078 21552 ts_tablet_manager.cc:1397] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 04:43:42.394727 21551 raft_consensus.cc:348] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1820e7a19bc49d6a1a4aace9570373c" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39775 } } peers { permanent_uuid: "73aad0d4724a444283b5ef30dac489a2" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 35419 } } peers { permanent_uuid: "eb9501d3b9d2405b9cafec880f182961" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39505 } }
I20260806 04:43:42.394977 21551 raft_consensus.cc:374] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:42.395031 21551 raft_consensus.cc:729] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1820e7a19bc49d6a1a4aace9570373c, State: Initialized, Role: FOLLOWER
I20260806 04:43:42.395568 21551 consensus_queue.cc:260] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1820e7a19bc49d6a1a4aace9570373c" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39775 } } peers { permanent_uuid: "73aad0d4724a444283b5ef30dac489a2" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 35419 } } peers { permanent_uuid: "eb9501d3b9d2405b9cafec880f182961" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39505 } }
I20260806 04:43:42.395514 21549 raft_consensus.cc:348] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1820e7a19bc49d6a1a4aace9570373c" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39775 } } peers { permanent_uuid: "73aad0d4724a444283b5ef30dac489a2" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 35419 } } peers { permanent_uuid: "eb9501d3b9d2405b9cafec880f182961" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39505 } }
I20260806 04:43:42.395749 21549 raft_consensus.cc:374] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:42.395807 21549 raft_consensus.cc:729] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9501d3b9d2405b9cafec880f182961, State: Initialized, Role: FOLLOWER
I20260806 04:43:42.396231 21394 heartbeater.cc:503] Master 127.31.166.190:39599 was elected leader, sending a full tablet report...
I20260806 04:43:42.396239 21549 consensus_queue.cc:260] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1820e7a19bc49d6a1a4aace9570373c" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39775 } } peers { permanent_uuid: "73aad0d4724a444283b5ef30dac489a2" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 35419 } } peers { permanent_uuid: "eb9501d3b9d2405b9cafec880f182961" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39505 } }
I20260806 04:43:42.396636 21551 ts_tablet_manager.cc:1428] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 04:43:42.396950 21257 heartbeater.cc:503] Master 127.31.166.190:39599 was elected leader, sending a full tablet report...
I20260806 04:43:42.397184 21549 ts_tablet_manager.cc:1428] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260806 04:43:42.397542 21552 raft_consensus.cc:348] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1820e7a19bc49d6a1a4aace9570373c" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39775 } } peers { permanent_uuid: "73aad0d4724a444283b5ef30dac489a2" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 35419 } } peers { permanent_uuid: "eb9501d3b9d2405b9cafec880f182961" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39505 } }
I20260806 04:43:42.397765 21552 raft_consensus.cc:374] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:42.397989 21552 raft_consensus.cc:729] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73aad0d4724a444283b5ef30dac489a2, State: Initialized, Role: FOLLOWER
I20260806 04:43:42.398418 21552 consensus_queue.cc:260] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1820e7a19bc49d6a1a4aace9570373c" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39775 } } peers { permanent_uuid: "73aad0d4724a444283b5ef30dac489a2" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 35419 } } peers { permanent_uuid: "eb9501d3b9d2405b9cafec880f182961" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39505 } }
I20260806 04:43:42.399142 21532 heartbeater.cc:503] Master 127.31.166.190:39599 was elected leader, sending a full tablet report...
I20260806 04:43:42.399410 21552 ts_tablet_manager.cc:1428] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260806 04:43:42.434389 21258 tablet.cc:2369] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:43:42.578217 21533 tablet.cc:2369] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 04:43:42.645918 21395 tablet.cc:2369] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 04:43:42.677497 21558 raft_consensus.cc:482] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:43:42.677690 21558 raft_consensus.cc:504] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1820e7a19bc49d6a1a4aace9570373c" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39775 } } peers { permanent_uuid: "73aad0d4724a444283b5ef30dac489a2" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 35419 } } peers { permanent_uuid: "eb9501d3b9d2405b9cafec880f182961" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39505 } }
I20260806 04:43:42.678753 21558 leader_election.cc:288] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1820e7a19bc49d6a1a4aace9570373c (127.31.166.130:39775), eb9501d3b9d2405b9cafec880f182961 (127.31.166.129:39505)
I20260806 04:43:42.682078 21557 raft_consensus.cc:482] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 04:43:42.682400 21557 raft_consensus.cc:504] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1820e7a19bc49d6a1a4aace9570373c" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39775 } } peers { permanent_uuid: "73aad0d4724a444283b5ef30dac489a2" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 35419 } } peers { permanent_uuid: "eb9501d3b9d2405b9cafec880f182961" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39505 } }
I20260806 04:43:42.683346 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.679130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:39775 (local address 127.31.166.131:41841)
0806 04:43:42.679289 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:42.679306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:42.679423 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:42.679784 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:42.679790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:42.679817 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:42.679971 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.679982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.680928 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.680932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.682064 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.682077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.682891 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.682894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.682991 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.683012 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 04:43:42.683144 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 04:43:42.683217 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 04:43:42.683353 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.679201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:41841 (local address 127.31.166.130:39775)
0806 04:43:42.679339 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:42.679369 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:42.679387 (+ 18us) server_negotiation.cc:407] Connection header received
0806 04:43:42.679548 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:42.679551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:42.679602 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:42.679696 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:42.680107 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.680820 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.682200 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.682771 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.682895 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.682946 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:42.683155 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 04:43:42.683220 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:43:42.683881 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.679609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.131:44665 (local address 127.31.166.129:39505)
0806 04:43:42.679764 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:42.679770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:42.679933 (+ 163us) server_negotiation.cc:407] Connection header received
0806 04:43:42.680012 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:42.680018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:42.680090 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:42.680171 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:42.680468 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.681191 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.682499 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.683063 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.683223 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.683275 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:42.683678 (+ 403us) server_negotiation.cc:299] Negotiation successful
0806 04:43:42.683771 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 04:43:42.683884 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.679454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:39505 (local address 127.31.166.131:44665)
0806 04:43:42.679821 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:42.679835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:42.679931 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:42.680197 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:42.680199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:42.680217 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:42.680342 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.680349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.681360 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.681364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.682355 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.682365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.683376 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.683381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.683484 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.683519 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 04:43:42.683649 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 04:43:42.683719 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":131,"threads_started":1}
I20260806 04:43:42.684405 21346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b0602a09b544b3a9a573e04a3fef81" candidate_uuid: "73aad0d4724a444283b5ef30dac489a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1820e7a19bc49d6a1a4aace9570373c" is_pre_election: true
I20260806 04:43:42.684681 21207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b0602a09b544b3a9a573e04a3fef81" candidate_uuid: "73aad0d4724a444283b5ef30dac489a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb9501d3b9d2405b9cafec880f182961" is_pre_election: true
I20260806 04:43:42.684749 21346 raft_consensus.cc:2454] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73aad0d4724a444283b5ef30dac489a2 in term 0.
I20260806 04:43:42.684883 21207 raft_consensus.cc:2454] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73aad0d4724a444283b5ef30dac489a2 in term 0.
I20260806 04:43:42.685153 21557 leader_election.cc:288] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1820e7a19bc49d6a1a4aace9570373c (127.31.166.130:39775), 73aad0d4724a444283b5ef30dac489a2 (127.31.166.131:35419)
I20260806 04:43:42.685276 21415 leader_election.cc:302] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73aad0d4724a444283b5ef30dac489a2, f1820e7a19bc49d6a1a4aace9570373c; no voters:
I20260806 04:43:42.685552 21558 raft_consensus.cc:2781] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 04:43:42.685702 21558 raft_consensus.cc:482] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 04:43:42.685814 21558 raft_consensus.cc:3037] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:42.687043 21558 raft_consensus.cc:504] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1820e7a19bc49d6a1a4aace9570373c" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39775 } } peers { permanent_uuid: "73aad0d4724a444283b5ef30dac489a2" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 35419 } } peers { permanent_uuid: "eb9501d3b9d2405b9cafec880f182961" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39505 } }
I20260806 04:43:42.687870 21346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b0602a09b544b3a9a573e04a3fef81" candidate_uuid: "73aad0d4724a444283b5ef30dac489a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1820e7a19bc49d6a1a4aace9570373c"
I20260806 04:43:42.688001 21346 raft_consensus.cc:3037] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:42.688046 21207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b0602a09b544b3a9a573e04a3fef81" candidate_uuid: "73aad0d4724a444283b5ef30dac489a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb9501d3b9d2405b9cafec880f182961"
I20260806 04:43:42.688159 21207 raft_consensus.cc:3037] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:42.688988 21545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.684169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:50961 (local address 127.31.166.130:39775)
0806 04:43:42.684274 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:42.684279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:42.684899 (+ 620us) server_negotiation.cc:407] Connection header received
0806 04:43:42.685156 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:42.685159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:42.685200 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:42.685243 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:42.685640 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.686310 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.687803 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.688379 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.688457 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.688496 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:42.688796 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 04:43:42.688872 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 04:43:42.689344 21560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.684074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.130:39775 (local address 127.31.166.129:50961)
0806 04:43:42.684480 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:42.684494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:42.685026 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:42.685402 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:42.685405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:42.685426 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:42.685509 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.685515 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.686437 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.686440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.687665 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.687675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.688584 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.688587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.688672 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.688693 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 04:43:42.689108 (+ 415us) client_negotiation.cc:240] Negotiation successful
0806 04:43:42.689180 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":80,"threads_started":1}
I20260806 04:43:42.689433 21207 raft_consensus.cc:2454] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73aad0d4724a444283b5ef30dac489a2 in term 1.
I20260806 04:43:42.689431 21346 raft_consensus.cc:2454] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73aad0d4724a444283b5ef30dac489a2 in term 1.
I20260806 04:43:42.689528 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.685001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.166.129:60517 (local address 127.31.166.131:35419)
0806 04:43:42.685116 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:42.685122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:42.685602 (+ 480us) server_negotiation.cc:407] Connection header received
0806 04:43:42.685768 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:42.685771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:42.685811 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:42.685854 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 04:43:42.686218 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.686907 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.688303 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.688870 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.688950 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.688988 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:42.689329 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 04:43:42.689422 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 04:43:42.689851 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.684625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.131:35419 (local address 127.31.166.129:60517)
0806 04:43:42.685530 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:42.685544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:43:42.685662 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:43:42.685977 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:43:42.685979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:43:42.685995 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 04:43:42.686107 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.686113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.687043 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.687046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.688174 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:43:42.688183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.689097 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.689102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:43:42.689195 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.689214 (+ 19us) client_negotiation.cc:769] Sending connection context
0806 04:43:42.689621 (+ 407us) client_negotiation.cc:240] Negotiation successful
0806 04:43:42.689719 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":112,"threads_started":1}
I20260806 04:43:42.689939 21415 leader_election.cc:302] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73aad0d4724a444283b5ef30dac489a2, eb9501d3b9d2405b9cafec880f182961; no voters:
I20260806 04:43:42.690167 21346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b0602a09b544b3a9a573e04a3fef81" candidate_uuid: "eb9501d3b9d2405b9cafec880f182961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1820e7a19bc49d6a1a4aace9570373c" is_pre_election: true
I20260806 04:43:42.690356 21562 raft_consensus.cc:2781] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:42.690387 21346 raft_consensus.cc:2379] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb9501d3b9d2405b9cafec880f182961 in current term 1: Already voted for candidate 73aad0d4724a444283b5ef30dac489a2 in this term.
I20260806 04:43:42.690591 21562 raft_consensus.cc:686] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [term 1 LEADER]: Becoming Leader. State: Replica: 73aad0d4724a444283b5ef30dac489a2, State: Running, Role: LEADER
I20260806 04:43:42.690649 21558 leader_election.cc:288] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [CANDIDATE]: Term 1 election: Requested vote from peers f1820e7a19bc49d6a1a4aace9570373c (127.31.166.130:39775), eb9501d3b9d2405b9cafec880f182961 (127.31.166.129:39505)
I20260806 04:43:42.690819 21484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b0602a09b544b3a9a573e04a3fef81" candidate_uuid: "eb9501d3b9d2405b9cafec880f182961" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73aad0d4724a444283b5ef30dac489a2" is_pre_election: true
I20260806 04:43:42.690961 21562 consensus_queue.cc:237] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1820e7a19bc49d6a1a4aace9570373c" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39775 } } peers { permanent_uuid: "73aad0d4724a444283b5ef30dac489a2" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 35419 } } peers { permanent_uuid: "eb9501d3b9d2405b9cafec880f182961" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39505 } }
I20260806 04:43:42.691417 21138 leader_election.cc:302] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb9501d3b9d2405b9cafec880f182961; no voters: 73aad0d4724a444283b5ef30dac489a2, f1820e7a19bc49d6a1a4aace9570373c
I20260806 04:43:42.691699 21557 raft_consensus.cc:2726] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 04:43:42.694324 21060 catalog_manager.cc:5495] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73aad0d4724a444283b5ef30dac489a2 (127.31.166.131). New cstate: current_term: 1 leader_uuid: "73aad0d4724a444283b5ef30dac489a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1820e7a19bc49d6a1a4aace9570373c" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73aad0d4724a444283b5ef30dac489a2" member_type: VOTER last_known_addr { host: "127.31.166.131" port: 35419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb9501d3b9d2405b9cafec880f182961" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39505 } health_report { overall_health: UNKNOWN } } }
I20260806 04:43:42.738698 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.735198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.31.166.131:35419)
0806 04:43:42.735341 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:42.735350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:42.735365 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:43:42.735479 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:42.735481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:42.735536 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:42.735624 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:43:42.735905 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.736409 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.737267 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:42.737497 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:42.737573 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:42.737847 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:43:42.738267 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:43:42.738376 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:42.738464 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 04:43:42.738510 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:43:42.744954 21346 raft_consensus.cc:1261] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c [term 1 FOLLOWER]: Refusing update from remote peer 73aad0d4724a444283b5ef30dac489a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:43:42.744995 21207 raft_consensus.cc:1261] T 90b0602a09b544b3a9a573e04a3fef81 P eb9501d3b9d2405b9cafec880f182961 [term 1 FOLLOWER]: Refusing update from remote peer 73aad0d4724a444283b5ef30dac489a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:43:42.745780 21563 consensus_queue.cc:1035] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1820e7a19bc49d6a1a4aace9570373c" member_type: VOTER last_known_addr { host: "127.31.166.130" port: 39775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:43:42.746008 21562 consensus_queue.cc:1035] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb9501d3b9d2405b9cafec880f182961" member_type: VOTER last_known_addr { host: "127.31.166.129" port: 39505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:43:42.754037 21566 mvcc.cc:205] Tried to move back new op lower bound from 7315420867555287040 to 7315420867345076224. Current Snapshot: MvccSnapshot[applied={T|T < 7315420867555287040}]
I20260806 04:43:42.758347 21569 mvcc.cc:205] Tried to move back new op lower bound from 7315420867555287040 to 7315420867345076224. Current Snapshot: MvccSnapshot[applied={T|T < 7315420867555287040 or (T in {7315420867555287040})}]
I20260806 04:43:42.763568 21060 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:43:42.763747 21060 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 04:43:42.766346 21060 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad2afaa1d1e14ef1b345ee7854f7a845: Sending DeleteTablet for 3 replicas of tablet 90b0602a09b544b3a9a573e04a3fef81
I20260806 04:43:42.767309 21323 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b0602a09b544b3a9a573e04a3fef81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:42 UTC) from {username='slave'} at 127.0.0.1:52132
I20260806 04:43:42.767304 21462 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b0602a09b544b3a9a573e04a3fef81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 04:43:42 UTC) from {username='slave'} at 127.0.0.1:46370
I20260806 04:43:42.767828 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 21123
I20260806 04:43:42.768050 21577 tablet_replica.cc:330] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2: stopping tablet replica
I20260806 04:43:42.767849 21578 tablet_replica.cc:330] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c: stopping tablet replica
I20260806 04:43:42.768430 21577 raft_consensus.cc:2229] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 04:43:42.768707 21578 raft_consensus.cc:2229] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 04:43:42.769011 21578 raft_consensus.cc:2258] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:43:42.769353 21577 raft_consensus.cc:2258] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 04:43:42.771528 21577 ts_tablet_manager.cc:1905] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:43:42.772840 21578 ts_tablet_manager.cc:1905] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 04:43:42.774544 21577 ts_tablet_manager.cc:1918] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:43:42.774683 21577 log.cc:1196] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-2/wal/wals/90b0602a09b544b3a9a573e04a3fef81
I20260806 04:43:42.775051 21577 ts_tablet_manager.cc:1939] T 90b0602a09b544b3a9a573e04a3fef81 P 73aad0d4724a444283b5ef30dac489a2: Deleting consensus metadata
I20260806 04:43:42.775888 21578 ts_tablet_manager.cc:1918] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 04:43:42.776026 21578 log.cc:1196] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c: Deleting WAL directory at /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785991160137184-32410-0/minicluster-data/ts-1/wal/wals/90b0602a09b544b3a9a573e04a3fef81
I20260806 04:43:42.776410 21578 ts_tablet_manager.cc:1939] T 90b0602a09b544b3a9a573e04a3fef81 P f1820e7a19bc49d6a1a4aace9570373c: Deleting consensus metadata
I20260806 04:43:42.776454 21043 catalog_manager.cc:4841] TS 73aad0d4724a444283b5ef30dac489a2 (127.31.166.131:35419): tablet 90b0602a09b544b3a9a573e04a3fef81 (table test-table [id=b03dda6b966f42bb94504fd510915d33]) successfully deleted
I20260806 04:43:42.777506 21044 catalog_manager.cc:4841] TS f1820e7a19bc49d6a1a4aace9570373c (127.31.166.130:39775): tablet 90b0602a09b544b3a9a573e04a3fef81 (table test-table [id=b03dda6b966f42bb94504fd510915d33]) successfully deleted
W20260806 04:43:42.778203 21044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.166.129:39505 (error 108)
I20260806 04:43:42.778451 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 21261
I20260806 04:43:42.779897 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:42.779547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.129:39505 (local address 127.0.0.1:53766)
0806 04:43:42.779745 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 04:43:42.779818 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.129:39505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260806 04:43:42.780084 21044 catalog_manager.cc:4584] TS eb9501d3b9d2405b9cafec880f182961 (127.31.166.129:39505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90b0602a09b544b3a9a573e04a3fef81: Network error: Client connection negotiation failed: client connection to 127.31.166.129:39505: connect: Connection refused (error 111)
I20260806 04:43:42.787493 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 21397
I20260806 04:43:42.794126 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 21027
2026-08-06T04:43:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1297 ms)
[----------] 6 tests from EncryptionPolicyTest (11488 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:42 dist-test-slave-vkjp krb5kdc[21582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:42 dist-test-slave-vkjp krb5kdc[21582](info): set up 2 sockets
Aug 06 04:43:42 dist-test-slave-vkjp krb5kdc[21582](info): commencing operation
krb5kdc: starting...
W20260806 04:43:43.873150 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:43:43 dist-test-slave-vkjp krb5kdc[21582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:43:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:43:44.002866 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:44.113730 21596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:44.114063 21596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:44.114156 21596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:44.118613 21596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:44.118709 21596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:44.118747 21596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:44.118805 21596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:44.118844 21596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:44.124192 21596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44953
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:44.125288 21596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:44.126329 21596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:44.134850 21602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:44.134912 21601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:44.134860 21604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:44.135202 21596 server_base.cc:1034] running on GCE node
I20260806 04:43:44.135862 21596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:44.136960 21596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:44.138155 21596 hybrid_clock.cc:648] HybridClock initialized: now 1785991424138145 us; error 31 us; skew 500 ppm
Aug 06 04:43:44 dist-test-slave-vkjp krb5kdc[21582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:44.141042 21596 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:43:44.141957 21596 webserver.cc:460] Webserver started at http://127.0.0.1:34863/ using document root <none> and password file <none>
I20260806 04:43:44.142508 21596 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:44.142599 21596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:44.142832 21596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:44.144893 21596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "d41c7fbbefd74cd798b470eac4ab1a9e"
format_stamp: "Formatted at 2026-08-06 04:43:44 on dist-test-slave-vkjp"
server_key: "0a06fdcf29a3a081cd4b49b1a05ef913"
server_key_iv: "e5c3a5b4149baf6bf2898316d26d7b16"
server_key_version: "encryptionkey@0"
I20260806 04:43:44.145473 21596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "d41c7fbbefd74cd798b470eac4ab1a9e"
format_stamp: "Formatted at 2026-08-06 04:43:44 on dist-test-slave-vkjp"
server_key: "0a06fdcf29a3a081cd4b49b1a05ef913"
server_key_iv: "e5c3a5b4149baf6bf2898316d26d7b16"
server_key_version: "encryptionkey@0"
I20260806 04:43:44.149231 21596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:43:44.151818 21611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:44.153061 21596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:43:44.153218 21596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "d41c7fbbefd74cd798b470eac4ab1a9e"
format_stamp: "Formatted at 2026-08-06 04:43:44 on dist-test-slave-vkjp"
server_key: "0a06fdcf29a3a081cd4b49b1a05ef913"
server_key_iv: "e5c3a5b4149baf6bf2898316d26d7b16"
server_key_version: "encryptionkey@0"
I20260806 04:43:44.153332 21596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:44.184085 21596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:44.187759 21596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:44.188006 21596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:44.195667 21596 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44953
I20260806 04:43:44.195685 21665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44953 every 8 connection(s)
I20260806 04:43:44.196655 21596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:44.199570 21666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:44.200469 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 21596
I20260806 04:43:44.200651 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:44.201128 32410 external_mini_cluster.cc:1428] Setting key 202cd7e503898aabe761639b8a74d339
I20260806 04:43:44.206094 21666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e: Bootstrap starting.
I20260806 04:43:44.208452 21666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:44.209460 21666 log.cc:824] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e: Log is configured to *not* fsync() on all Append() calls
Aug 06 04:43:44 dist-test-slave-vkjp krb5kdc[21582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 04:43:44.211669 21666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e: No bootstrap required, opened a new log
I20260806 04:43:44.215386 21666 raft_consensus.cc:348] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41c7fbbefd74cd798b470eac4ab1a9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44953 } }
I20260806 04:43:44.215651 21666 raft_consensus.cc:374] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:44.215749 21666 raft_consensus.cc:729] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41c7fbbefd74cd798b470eac4ab1a9e, State: Initialized, Role: FOLLOWER
I20260806 04:43:44.216032 21670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:44.202690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40629 (local address 127.0.0.1:44953)
0806 04:43:44.203217 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:44.203232 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:44.203268 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:43:44.203883 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:44.203899 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:44.204211 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:44.204538 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:44.205364 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:44.206374 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:44.207277 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:44.207743 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:44.207928 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:44.211193 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:44.211198 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:44.211210 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:44.211237 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:44.212963 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:44.213488 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:44.213498 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:44.213502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:44.213610 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:44.213967 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:44.213976 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:44.213978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:44.214355 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:44.214592 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:44.214883 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 04:43:44.215173 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":164,"threads_started":1}
I20260806 04:43:44.216310 21666 consensus_queue.cc:260] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41c7fbbefd74cd798b470eac4ab1a9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44953 } }
I20260806 04:43:44.216485 21666 raft_consensus.cc:388] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:44.216557 21666 raft_consensus.cc:482] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:44.216657 21666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:44.217787 21666 raft_consensus.cc:504] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41c7fbbefd74cd798b470eac4ab1a9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44953 } }
I20260806 04:43:44.218174 21666 leader_election.cc:302] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d41c7fbbefd74cd798b470eac4ab1a9e; no voters:
I20260806 04:43:44.218556 21666 leader_election.cc:288] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:44.218670 21672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:44.218957 21672 raft_consensus.cc:686] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [term 1 LEADER]: Becoming Leader. State: Replica: d41c7fbbefd74cd798b470eac4ab1a9e, State: Running, Role: LEADER
I20260806 04:43:44.219380 21672 consensus_queue.cc:237] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41c7fbbefd74cd798b470eac4ab1a9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44953 } }
I20260806 04:43:44.219537 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:43:44.219998 21666 sys_catalog.cc:564] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:44.220721 21672 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d41c7fbbefd74cd798b470eac4ab1a9e. Latest consensus state: current_term: 1 leader_uuid: "d41c7fbbefd74cd798b470eac4ab1a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41c7fbbefd74cd798b470eac4ab1a9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44953 } } }
I20260806 04:43:44.220855 21672 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:44.221397 21673 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d41c7fbbefd74cd798b470eac4ab1a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41c7fbbefd74cd798b470eac4ab1a9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44953 } } }
I20260806 04:43:44.221510 21673 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e [sys.catalog]: This master's current role is: LEADER
W20260806 04:43:44.225128 21693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:43:44.225236 21693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:43:44.225315 21683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:44.226150 21683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:44.232194 21683 catalog_manager.cc:1347] Generated new cluster ID: 3f0e5c8660ca436285dc43413491680b
I20260806 04:43:44.232265 21683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:44.233589 21670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:44.223491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60627 (local address 127.0.0.1:44953)
0806 04:43:44.223623 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:44.223629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:44.224051 (+ 422us) server_negotiation.cc:407] Connection header received
0806 04:43:44.224153 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:44.224156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:44.224206 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:44.224294 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:44.225221 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:44.225958 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:44.227225 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:44.227543 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:44.227670 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:44.229594 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:44.229598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:44.229601 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:44.229629 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:44.231490 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:44.231998 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:44.232002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:44.232003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:44.232112 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:44.232547 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:44.232551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:44.232554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:44.232811 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:44.232979 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:44.233304 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 04:43:44.233419 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:43:44.251946 21683 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:44.252926 21683 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:44.257110 21683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d41c7fbbefd74cd798b470eac4ab1a9e: Generated new TSK 0
I20260806 04:43:44.257855 21683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:43:44.263942 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 21596
W20260806 04:43:44.270653 21580 connection.cc:535] client connection to 127.0.0.1:44953 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44953): Connection reset by peer (error 104)
2026-08-06T04:43:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:44 dist-test-slave-vkjp krb5kdc[21700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:44 dist-test-slave-vkjp krb5kdc[21700](info): set up 2 sockets
Aug 06 04:43:44 dist-test-slave-vkjp krb5kdc[21700](info): commencing operation
krb5kdc: starting...
W20260806 04:43:45.330718 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:43:45 dist-test-slave-vkjp krb5kdc[21700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:43:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:43:45.462167 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:45.571893 21714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:45.572178 21714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:45.572220 21714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:45.576121 21714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:45.576177 21714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:45.576197 21714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:45.576213 21714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:45.576229 21714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:45.581563 21714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36633
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:45.582566 21714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:45.583580 21714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:45.592865 21722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:45.593067 21714 server_base.cc:1034] running on GCE node
W20260806 04:43:45.593128 21721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:45.592870 21724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:45.593943 21714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:45.595464 21714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:45.596664 21714 hybrid_clock.cc:648] HybridClock initialized: now 1785991425596628 us; error 57 us; skew 500 ppm
Aug 06 04:43:45 dist-test-slave-vkjp krb5kdc[21700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:45.602984 21714 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:43:45.604022 21714 webserver.cc:460] Webserver started at http://127.0.0.1:38417/ using document root <none> and password file <none>
I20260806 04:43:45.604563 21714 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:45.604624 21714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:45.604799 21714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:45.606842 21714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "5b104beb7d094c4ca7644860b3271d7c"
format_stamp: "Formatted at 2026-08-06 04:43:45 on dist-test-slave-vkjp"
server_key: "fa704e0093af93d058ac19a724478bb1"
server_key_iv: "b24672fc8513b83be011da2b895cac23"
server_key_version: "encryptionkey@0"
I20260806 04:43:45.607421 21714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "5b104beb7d094c4ca7644860b3271d7c"
format_stamp: "Formatted at 2026-08-06 04:43:45 on dist-test-slave-vkjp"
server_key: "fa704e0093af93d058ac19a724478bb1"
server_key_iv: "b24672fc8513b83be011da2b895cac23"
server_key_version: "encryptionkey@0"
I20260806 04:43:45.611290 21714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:43:45.613729 21732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:45.615058 21714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:43:45.615267 21714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "5b104beb7d094c4ca7644860b3271d7c"
format_stamp: "Formatted at 2026-08-06 04:43:45 on dist-test-slave-vkjp"
server_key: "fa704e0093af93d058ac19a724478bb1"
server_key_iv: "b24672fc8513b83be011da2b895cac23"
server_key_version: "encryptionkey@0"
I20260806 04:43:45.615382 21714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:45.631515 21714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:45.635330 21714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:45.635592 21714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:45.643679 21714 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36633
I20260806 04:43:45.643762 21788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36633 every 8 connection(s)
I20260806 04:43:45.644742 21714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:45.647763 21789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:45.649597 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 21714
I20260806 04:43:45.649771 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:45.650092 32410 external_mini_cluster.cc:1428] Setting key d05a642ab985b9fa7286338d0e6da19b
I20260806 04:43:45.655284 21789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c: Bootstrap starting.
I20260806 04:43:45.658604 21789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:45.659623 21789 log.cc:824] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:45.661006 21792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:45.651725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47799 (local address 127.0.0.1:36633)
0806 04:43:45.652238 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:45.652262 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:45.652307 (+ 45us) server_negotiation.cc:407] Connection header received
0806 04:43:45.653125 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:45.653149 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:45.653546 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:45.653878 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:45.654338 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:45.655420 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:45.656773 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:45.657312 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:45.657505 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:45.658551 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:45.658560 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:45.658575 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:45.659196 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:45.659265 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:45.659293 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:45.659411 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:45.659645 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 04:43:45.659830 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":178,"threads_started":1}
I20260806 04:43:45.662375 21789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c: No bootstrap required, opened a new log
I20260806 04:43:45.664741 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:43:45.668443 21789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b104beb7d094c4ca7644860b3271d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36633 } }
I20260806 04:43:45.668813 21789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:45.668903 21789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b104beb7d094c4ca7644860b3271d7c, State: Initialized, Role: FOLLOWER
I20260806 04:43:45.669459 21789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b104beb7d094c4ca7644860b3271d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36633 } }
I20260806 04:43:45.669595 21789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:45.669684 21789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:45.669808 21789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:45.670855 21789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b104beb7d094c4ca7644860b3271d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36633 } }
I20260806 04:43:45.671304 21789 leader_election.cc:302] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b104beb7d094c4ca7644860b3271d7c; no voters:
I20260806 04:43:45.671697 21789 leader_election.cc:288] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:45.671902 21799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:45.672245 21799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [term 1 LEADER]: Becoming Leader. State: Replica: 5b104beb7d094c4ca7644860b3271d7c, State: Running, Role: LEADER
I20260806 04:43:45.672649 21792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:45.667924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36493 (local address 127.0.0.1:36633)
0806 04:43:45.668081 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:45.668085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:45.668260 (+ 175us) server_negotiation.cc:407] Connection header received
0806 04:43:45.668436 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:45.668439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:45.668489 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:45.668570 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:45.668983 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:45.669759 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:45.670651 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:45.670927 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:45.671046 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:45.672054 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:45.672060 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:45.672065 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:45.672080 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:45.672115 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:45.672129 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:45.672204 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:45.672473 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 04:43:45.672515 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 04:43:45.672662 21799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b104beb7d094c4ca7644860b3271d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36633 } }
I20260806 04:43:45.673163 21789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:45.675207 21800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b104beb7d094c4ca7644860b3271d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b104beb7d094c4ca7644860b3271d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36633 } } }
I20260806 04:43:45.675701 21800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:45.675415 21801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b104beb7d094c4ca7644860b3271d7c. Latest consensus state: current_term: 1 leader_uuid: "5b104beb7d094c4ca7644860b3271d7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b104beb7d094c4ca7644860b3271d7c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36633 } } }
I20260806 04:43:45.675987 21801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c [sys.catalog]: This master's current role is: LEADER
W20260806 04:43:45.677958 21813 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:43:45.678058 21813 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:43:45.678133 21815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:45.678818 21815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:45.684435 21815 catalog_manager.cc:1347] Generated new cluster ID: b2f54d2a92aa440a8a8bbefa8e750e24
I20260806 04:43:45.684516 21815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:45.728864 21815 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:45.730156 21815 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:45.740674 21815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b104beb7d094c4ca7644860b3271d7c: Generated new TSK 0
I20260806 04:43:45.741578 21815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:43:45.754901 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 21714
W20260806 04:43:45.761402 21698 connection.cc:535] client connection to 127.0.0.1:36633 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36633): Connection reset by peer (error 104)
2026-08-06T04:43:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1490 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:45 dist-test-slave-vkjp krb5kdc[21820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:45 dist-test-slave-vkjp krb5kdc[21820](info): set up 2 sockets
Aug 06 04:43:45 dist-test-slave-vkjp krb5kdc[21820](info): commencing operation
krb5kdc: starting...
W20260806 04:43:46.841910 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:43:46 dist-test-slave-vkjp krb5kdc[21820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:43:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:43:46.971009 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:47.084568 21835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:47.084859 21835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:47.084934 21835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:47.089071 21835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:47.089159 21835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:47.089202 21835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:47.089269 21835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:47.089325 21835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:47.094633 21835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46625
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:47.095745 21835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:47.096760 21835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:47.105823 21844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:47.105796 21840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:47.105786 21841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:47.105893 21835 server_base.cc:1034] running on GCE node
I20260806 04:43:47.106570 21835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:47.107769 21835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:47.108951 21835 hybrid_clock.cc:648] HybridClock initialized: now 1785991427108942 us; error 36 us; skew 500 ppm
Aug 06 04:43:47 dist-test-slave-vkjp krb5kdc[21820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:47.111752 21835 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:43:47.112630 21835 webserver.cc:460] Webserver started at http://127.0.0.1:43045/ using document root <none> and password file <none>
I20260806 04:43:47.113154 21835 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:47.113214 21835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:47.113384 21835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:47.115434 21835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "13675121ec6d4046a94521d280fbcb5a"
format_stamp: "Formatted at 2026-08-06 04:43:47 on dist-test-slave-vkjp"
server_key: "eff53096be48c5f6420d19f72c887c3b"
server_key_iv: "e291a389e8fec36750bb76804b1e77e6"
server_key_version: "encryptionkey@0"
I20260806 04:43:47.115944 21835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "13675121ec6d4046a94521d280fbcb5a"
format_stamp: "Formatted at 2026-08-06 04:43:47 on dist-test-slave-vkjp"
server_key: "eff53096be48c5f6420d19f72c887c3b"
server_key_iv: "e291a389e8fec36750bb76804b1e77e6"
server_key_version: "encryptionkey@0"
I20260806 04:43:47.119761 21835 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 04:43:47.122179 21851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:47.123575 21835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:43:47.123720 21835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "13675121ec6d4046a94521d280fbcb5a"
format_stamp: "Formatted at 2026-08-06 04:43:47 on dist-test-slave-vkjp"
server_key: "eff53096be48c5f6420d19f72c887c3b"
server_key_iv: "e291a389e8fec36750bb76804b1e77e6"
server_key_version: "encryptionkey@0"
I20260806 04:43:47.123829 21835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:47.128484 21835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:47.131837 21835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:47.132042 21835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:47.139420 21835 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46625
I20260806 04:43:47.139458 21909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46625 every 8 connection(s)
I20260806 04:43:47.140404 21835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:47.143323 21910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:47.148639 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 21835
I20260806 04:43:47.148810 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:47.148968 21910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a: Bootstrap starting.
I20260806 04:43:47.149165 32410 external_mini_cluster.cc:1428] Setting key c5df1abc9462efdc682733dd06a25611
I20260806 04:43:47.151571 21910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:47.152321 21910 log.cc:824] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:47.154495 21910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a: No bootstrap required, opened a new log
I20260806 04:43:47.154642 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:47.150615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57775 (local address 127.0.0.1:46625)
0806 04:43:47.151039 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:47.151062 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:47.151151 (+ 89us) server_negotiation.cc:407] Connection header received
0806 04:43:47.151864 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:47.151883 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:47.152158 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:47.152475 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:47.153315 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:47.153319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:47.153330 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:47.153340 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:47.153398 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:47.153418 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:47.153503 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:47.153687 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 04:43:47.153888 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":149,"threads_started":1}
I20260806 04:43:47.157559 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:43:47.158455 21910 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13675121ec6d4046a94521d280fbcb5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } }
I20260806 04:43:47.158738 21910 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:47.158807 21910 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13675121ec6d4046a94521d280fbcb5a, State: Initialized, Role: FOLLOWER
I20260806 04:43:47.159350 21910 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13675121ec6d4046a94521d280fbcb5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } }
I20260806 04:43:47.159519 21910 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:47.159593 21910 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:47.159706 21910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:47.160720 21910 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13675121ec6d4046a94521d280fbcb5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } }
I20260806 04:43:47.161072 21910 leader_election.cc:302] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13675121ec6d4046a94521d280fbcb5a; no voters:
I20260806 04:43:47.161373 21910 leader_election.cc:288] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:47.161459 21920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:47.161671 21920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [term 1 LEADER]: Becoming Leader. State: Replica: 13675121ec6d4046a94521d280fbcb5a, State: Running, Role: LEADER
I20260806 04:43:47.161815 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:47.159925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35725 (local address 127.0.0.1:46625)
0806 04:43:47.160035 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:47.160039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:47.160241 (+ 202us) server_negotiation.cc:407] Connection header received
0806 04:43:47.160417 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:47.160420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:47.160459 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:47.160543 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:47.161322 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:47.161325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:47.161327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:47.161338 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:47.161377 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:47.161392 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:47.161472 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:47.161643 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 04:43:47.161688 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 04:43:47.162048 21920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13675121ec6d4046a94521d280fbcb5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } }
I20260806 04:43:47.162745 21910 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:47.163291 21922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13675121ec6d4046a94521d280fbcb5a. Latest consensus state: current_term: 1 leader_uuid: "13675121ec6d4046a94521d280fbcb5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13675121ec6d4046a94521d280fbcb5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } } }
I20260806 04:43:47.163421 21922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:47.163705 21921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13675121ec6d4046a94521d280fbcb5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13675121ec6d4046a94521d280fbcb5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46625 } } }
I20260806 04:43:47.163829 21921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:47.163988 21925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:47.167147 21925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:47.172938 21925 catalog_manager.cc:1347] Generated new cluster ID: b306dd00b1544eab8003369597d8c2cf
I20260806 04:43:47.173022 21925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:47.211107 21925 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:47.211493 21925 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:47.220872 21925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13675121ec6d4046a94521d280fbcb5a: Generated new TSK 0
I20260806 04:43:47.221647 21925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:43:47.229488 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 21835
2026-08-06T04:43:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1475 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:47 dist-test-slave-vkjp krb5kdc[21942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:47 dist-test-slave-vkjp krb5kdc[21942](info): set up 2 sockets
Aug 06 04:43:47 dist-test-slave-vkjp krb5kdc[21942](info): commencing operation
krb5kdc: starting...
W20260806 04:43:48.334147 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:43:48 dist-test-slave-vkjp krb5kdc[21942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:43:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:43:48.467278 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44567
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:48.581175 21956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:48.581516 21956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:48.581601 21956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:48.585690 21956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:48.585810 21956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:48.585850 21956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:48.585896 21956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:48.585922 21956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:48.591696 21956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:48.592906 21956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:48.593990 21956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:48.603166 21962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:48.603166 21961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:48.603200 21964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:48.603190 21956 server_base.cc:1034] running on GCE node
I20260806 04:43:48.604135 21956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:48.605347 21956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:48.606546 21956 hybrid_clock.cc:648] HybridClock initialized: now 1785991428606532 us; error 30 us; skew 500 ppm
Aug 06 04:43:48 dist-test-slave-vkjp krb5kdc[21942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:48.609500 21956 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:43:48.610440 21956 webserver.cc:460] Webserver started at http://127.0.0.1:37735/ using document root <none> and password file <none>
I20260806 04:43:48.610975 21956 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:48.611034 21956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:48.611275 21956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:48.613354 21956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "ce0e204921c245438b3c9e64064a9c99"
format_stamp: "Formatted at 2026-08-06 04:43:48 on dist-test-slave-vkjp"
server_key: "4d70587e43ec401570a6ffb400ea3404"
server_key_iv: "abf567a5efc55200cab061b11e92d1fd"
server_key_version: "encryptionkey@0"
I20260806 04:43:48.613902 21956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "ce0e204921c245438b3c9e64064a9c99"
format_stamp: "Formatted at 2026-08-06 04:43:48 on dist-test-slave-vkjp"
server_key: "4d70587e43ec401570a6ffb400ea3404"
server_key_iv: "abf567a5efc55200cab061b11e92d1fd"
server_key_version: "encryptionkey@0"
I20260806 04:43:48.617947 21956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 04:43:48.620741 21971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:48.622202 21956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:43:48.622346 21956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "ce0e204921c245438b3c9e64064a9c99"
format_stamp: "Formatted at 2026-08-06 04:43:48 on dist-test-slave-vkjp"
server_key: "4d70587e43ec401570a6ffb400ea3404"
server_key_iv: "abf567a5efc55200cab061b11e92d1fd"
server_key_version: "encryptionkey@0"
I20260806 04:43:48.622520 21956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:48.650482 21956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:48.654335 21956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:48.654515 21956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:48.662113 21956 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44567
I20260806 04:43:48.662139 22026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44567 every 8 connection(s)
I20260806 04:43:48.663120 21956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:48.664609 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 21956
I20260806 04:43:48.664774 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:48.665177 32410 external_mini_cluster.cc:1428] Setting key 675a725469c66a3f5a8cd59e2ac01e2e
I20260806 04:43:48.666508 22027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:48.672287 22027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99: Bootstrap starting.
Aug 06 04:43:48 dist-test-slave-vkjp krb5kdc[21942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 04:43:48.674664 22027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:48.675515 22027 log.cc:824] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:48.677768 22027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99: No bootstrap required, opened a new log
I20260806 04:43:48.679870 22030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:48.666678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34359 (local address 127.0.0.1:44567)
0806 04:43:48.667199 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:48.667215 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:48.667252 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:43:48.667917 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:48.667935 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:48.668320 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:48.668662 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:48.669515 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:48.670516 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:48.671469 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:48.671958 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:48.672137 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:48.674711 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:48.674716 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:48.674730 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:48.674765 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:48.676816 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:48.677293 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:48.677306 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:48.677311 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:48.677426 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:48.677733 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:48.677738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:48.677740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:48.678254 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:48.678463 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:48.678776 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 04:43:48.679118 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":174,"threads_started":1}
I20260806 04:43:48.682104 22027 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e204921c245438b3c9e64064a9c99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } }
I20260806 04:43:48.682456 22027 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:48.682569 22027 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0e204921c245438b3c9e64064a9c99, State: Initialized, Role: FOLLOWER
I20260806 04:43:48.683173 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:43:48.683149 22027 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e204921c245438b3c9e64064a9c99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } }
I20260806 04:43:48.683357 22027 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:48.683457 22027 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:48.683568 22027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:48.684690 22027 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e204921c245438b3c9e64064a9c99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } }
I20260806 04:43:48.685075 22027 leader_election.cc:302] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce0e204921c245438b3c9e64064a9c99; no voters:
I20260806 04:43:48.685467 22027 leader_election.cc:288] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:48.685716 22037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:48.686275 22037 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [term 1 LEADER]: Becoming Leader. State: Replica: ce0e204921c245438b3c9e64064a9c99, State: Running, Role: LEADER
I20260806 04:43:48.686661 22037 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e204921c245438b3c9e64064a9c99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } }
I20260806 04:43:48.687023 22027 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:48.688546 22039 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce0e204921c245438b3c9e64064a9c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e204921c245438b3c9e64064a9c99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } } }
I20260806 04:43:48.688688 22039 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:48.689110 22040 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce0e204921c245438b3c9e64064a9c99. Latest consensus state: current_term: 1 leader_uuid: "ce0e204921c245438b3c9e64064a9c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce0e204921c245438b3c9e64064a9c99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44567 } } }
I20260806 04:43:48.689309 22040 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99 [sys.catalog]: This master's current role is: LEADER
W20260806 04:43:48.692616 22055 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:43:48.692790 22055 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:43:48.692890 22049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:48.693733 22049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:48.696043 22030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:48.685545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51613 (local address 127.0.0.1:44567)
0806 04:43:48.685696 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:48.685702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:48.685954 (+ 252us) server_negotiation.cc:407] Connection header received
0806 04:43:48.686153 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:48.686156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:48.686207 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:48.686254 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:48.687274 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:48.687909 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:48.689191 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:48.689539 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:48.689640 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:48.691614 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:48.691618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:48.691621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:48.691650 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:48.693497 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:48.694080 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:48.694084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:48.694086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:48.694181 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:48.694505 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:48.694508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:48.694510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:48.694756 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:48.694903 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:48.695715 (+ 812us) server_negotiation.cc:299] Negotiation successful
0806 04:43:48.695873 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:43:48.700431 22049 catalog_manager.cc:1347] Generated new cluster ID: 6c1fb37656b643bdbaa7001c38bf8339
I20260806 04:43:48.700584 22049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:48.714166 22049 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:48.715550 22049 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:48.724283 22049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce0e204921c245438b3c9e64064a9c99: Generated new TSK 0
I20260806 04:43:48.725091 22049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:43:48.730189 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 21956
2026-08-06T04:43:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1501 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:48 dist-test-slave-vkjp krb5kdc[22061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:48 dist-test-slave-vkjp krb5kdc[22061](info): set up 2 sockets
Aug 06 04:43:48 dist-test-slave-vkjp krb5kdc[22061](info): commencing operation
krb5kdc: starting...
W20260806 04:43:49.849730 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:43:49 dist-test-slave-vkjp krb5kdc[22061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:43:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:43:49.990106 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:50.109891 22075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:50.110157 22075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:50.110199 22075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:50.114274 22075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:50.114346 22075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:50.114365 22075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:50.114382 22075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:50.114396 22075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:50.119730 22075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:50.120783 22075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:50.121834 22075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:50.130484 22083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:50.130527 22081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:50.130507 22080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:50.131034 22075 server_base.cc:1034] running on GCE node
I20260806 04:43:50.131626 22075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:50.132711 22075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:50.133893 22075 hybrid_clock.cc:648] HybridClock initialized: now 1785991430133875 us; error 33 us; skew 500 ppm
Aug 06 04:43:50 dist-test-slave-vkjp krb5kdc[22061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:50.136891 22075 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:43:50.137861 22075 webserver.cc:460] Webserver started at http://127.0.0.1:38447/ using document root <none> and password file <none>
I20260806 04:43:50.138458 22075 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:50.138552 22075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:50.138811 22075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:50.140929 22075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "9796fcac507141a7a3ecd4b708e52c96"
format_stamp: "Formatted at 2026-08-06 04:43:50 on dist-test-slave-vkjp"
server_key: "1824fa1b8ddac69125309b870b337611"
server_key_iv: "b50169351b1baac129cb737ee5fa5edb"
server_key_version: "encryptionkey@0"
I20260806 04:43:50.141516 22075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "9796fcac507141a7a3ecd4b708e52c96"
format_stamp: "Formatted at 2026-08-06 04:43:50 on dist-test-slave-vkjp"
server_key: "1824fa1b8ddac69125309b870b337611"
server_key_iv: "b50169351b1baac129cb737ee5fa5edb"
server_key_version: "encryptionkey@0"
I20260806 04:43:50.145339 22075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:43:50.148059 22090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:50.149467 22075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 04:43:50.149621 22075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "9796fcac507141a7a3ecd4b708e52c96"
format_stamp: "Formatted at 2026-08-06 04:43:50 on dist-test-slave-vkjp"
server_key: "1824fa1b8ddac69125309b870b337611"
server_key_iv: "b50169351b1baac129cb737ee5fa5edb"
server_key_version: "encryptionkey@0"
I20260806 04:43:50.149747 22075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:50.164484 22075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:50.167953 22075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:50.168175 22075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:50.175312 22075 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34533
I20260806 04:43:50.175377 22144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34533 every 8 connection(s)
I20260806 04:43:50.176308 22075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:50.177245 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 22075
I20260806 04:43:50.177415 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:50.177711 32410 external_mini_cluster.cc:1428] Setting key 320ed031a7f0ecbb0f1ab1ad21195c3b
I20260806 04:43:50.179598 22145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:50.186101 22145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96: Bootstrap starting.
I20260806 04:43:50.186712 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:50.179054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51645 (local address 127.0.0.1:34533)
0806 04:43:50.179524 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:50.179539 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:50.179574 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:43:50.180238 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:50.180254 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:50.180581 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:50.180863 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:50.181257 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:50.182251 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:50.183490 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:50.183953 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:50.184071 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:50.184982 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:50.184987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:50.184999 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:50.185377 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:50.185438 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:50.185461 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:50.185537 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:50.185704 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 04:43:50.185941 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":145,"threads_started":1}
I20260806 04:43:50.188661 22145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:50.189522 22145 log.cc:824] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:50.189833 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:43:50.191975 22145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96: No bootstrap required, opened a new log
I20260806 04:43:50.196911 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:50.192216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40447 (local address 127.0.0.1:34533)
0806 04:43:50.192339 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:50.192344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:50.192553 (+ 209us) server_negotiation.cc:407] Connection header received
0806 04:43:50.192711 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:50.192715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:50.192767 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:50.192861 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:50.193134 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:50.194001 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:50.195282 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:50.195501 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:50.195584 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:50.196450 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:50.196453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:50.196455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:50.196466 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:50.196488 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:50.196497 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:50.196556 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:50.196665 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 04:43:50.196713 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:43:50.196856 22145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9796fcac507141a7a3ecd4b708e52c96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34533 } }
I20260806 04:43:50.197182 22145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:50.197245 22145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9796fcac507141a7a3ecd4b708e52c96, State: Initialized, Role: FOLLOWER
I20260806 04:43:50.197670 22145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9796fcac507141a7a3ecd4b708e52c96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34533 } }
I20260806 04:43:50.197824 22145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:50.197906 22145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:50.198007 22145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:50.198968 22145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9796fcac507141a7a3ecd4b708e52c96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34533 } }
I20260806 04:43:50.199347 22145 leader_election.cc:302] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9796fcac507141a7a3ecd4b708e52c96; no voters:
I20260806 04:43:50.199702 22145 leader_election.cc:288] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:50.199781 22155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:50.199995 22155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [term 1 LEADER]: Becoming Leader. State: Replica: 9796fcac507141a7a3ecd4b708e52c96, State: Running, Role: LEADER
I20260806 04:43:50.200363 22155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9796fcac507141a7a3ecd4b708e52c96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34533 } }
I20260806 04:43:50.201129 22145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:50.201620 22157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9796fcac507141a7a3ecd4b708e52c96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9796fcac507141a7a3ecd4b708e52c96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34533 } } }
I20260806 04:43:50.201745 22157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:50.202450 22156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9796fcac507141a7a3ecd4b708e52c96. Latest consensus state: current_term: 1 leader_uuid: "9796fcac507141a7a3ecd4b708e52c96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9796fcac507141a7a3ecd4b708e52c96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34533 } } }
I20260806 04:43:50.202605 22156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:50.203445 22160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:50.206513 22160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:50.211793 22160 catalog_manager.cc:1347] Generated new cluster ID: 88568115ba7f4ed9aa00a189fbde099c
I20260806 04:43:50.211869 22160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:50.232405 22160 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:50.233740 22160 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:50.242203 22160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9796fcac507141a7a3ecd4b708e52c96: Generated new TSK 0
I20260806 04:43:50.243026 22160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:43:50.252451 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 22075
2026-08-06T04:43:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1520 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:50 dist-test-slave-vkjp krb5kdc[22180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:50 dist-test-slave-vkjp krb5kdc[22180](info): set up 2 sockets
Aug 06 04:43:50 dist-test-slave-vkjp krb5kdc[22180](info): commencing operation
krb5kdc: starting...
W20260806 04:43:51.326195 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:43:51 dist-test-slave-vkjp krb5kdc[22180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:43:51.455013 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32911
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:51.566612 22195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:51.566931 22195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:51.567008 22195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:51.571038 22195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:51.571161 22195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:51.571201 22195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:51.571264 22195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:51.571302 22195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:51.576689 22195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:51.577836 22195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:51.578886 22195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:51.587389 22204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:51.587399 22202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:51.587725 22195 server_base.cc:1034] running on GCE node
W20260806 04:43:51.587761 22201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:51.588632 22195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:51.589745 22195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:51.590939 22195 hybrid_clock.cc:648] HybridClock initialized: now 1785991431590911 us; error 52 us; skew 500 ppm
Aug 06 04:43:51 dist-test-slave-vkjp krb5kdc[22180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:51.599850 22195 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:43:51.600839 22195 webserver.cc:460] Webserver started at http://127.0.0.1:33787/ using document root <none> and password file <none>
I20260806 04:43:51.601465 22195 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:51.601565 22195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:51.601814 22195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:51.604012 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "4c4ce07a334a473c9935125ab10f4953"
format_stamp: "Formatted at 2026-08-06 04:43:51 on dist-test-slave-vkjp"
server_key: "028d79d89e80ddc9aa5530beb0ff8786"
server_key_iv: "84458609d5a437f089659f3520d9cd9a"
server_key_version: "encryptionkey@0"
I20260806 04:43:51.604610 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "4c4ce07a334a473c9935125ab10f4953"
format_stamp: "Formatted at 2026-08-06 04:43:51 on dist-test-slave-vkjp"
server_key: "028d79d89e80ddc9aa5530beb0ff8786"
server_key_iv: "84458609d5a437f089659f3520d9cd9a"
server_key_version: "encryptionkey@0"
I20260806 04:43:51.608381 22195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 04:43:51.610808 22211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:51.612118 22195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:43:51.612268 22195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "4c4ce07a334a473c9935125ab10f4953"
format_stamp: "Formatted at 2026-08-06 04:43:51 on dist-test-slave-vkjp"
server_key: "028d79d89e80ddc9aa5530beb0ff8786"
server_key_iv: "84458609d5a437f089659f3520d9cd9a"
server_key_version: "encryptionkey@0"
I20260806 04:43:51.612386 22195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:51.617161 22195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:51.620607 22195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:51.620814 22195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:51.628336 22195 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32911
I20260806 04:43:51.628357 22265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32911 every 8 connection(s)
I20260806 04:43:51.629318 22195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:51.631681 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 22195
I20260806 04:43:51.631836 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:51.632155 32410 external_mini_cluster.cc:1428] Setting key 28a753f2b4aaf7e3807f1a949ad5adac
I20260806 04:43:51.632318 22266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:51.638016 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:51.633694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37653 (local address 127.0.0.1:32911)
0806 04:43:51.634121 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:51.634140 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:51.634183 (+ 43us) server_negotiation.cc:407] Connection header received
0806 04:43:51.635108 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:51.635133 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:51.635507 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:51.635886 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:51.636629 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:51.636634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:51.636648 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:51.636660 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:51.636722 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:51.636747 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:51.636886 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:51.636950 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 04:43:51.637101 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
I20260806 04:43:51.639799 22266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953: Bootstrap starting.
I20260806 04:43:51.640897 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:43:51.642242 22266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:51.643150 22266 log.cc:824] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:51.644773 22270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:51.643013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50785 (local address 127.0.0.1:32911)
0806 04:43:51.643160 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:51.643166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:51.643362 (+ 196us) server_negotiation.cc:407] Connection header received
0806 04:43:51.643486 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:51.643492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:51.643533 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:51.643638 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:51.644425 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:51.644428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:51.644431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:51.644442 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:51.644468 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:51.644481 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:51.644576 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:51.644630 (+ 54us) server_negotiation.cc:299] Negotiation successful
0806 04:43:51.644665 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:43:51.645627 22266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953: No bootstrap required, opened a new log
I20260806 04:43:51.649168 22266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4ce07a334a473c9935125ab10f4953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32911 } }
I20260806 04:43:51.649430 22266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:51.649490 22266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c4ce07a334a473c9935125ab10f4953, State: Initialized, Role: FOLLOWER
I20260806 04:43:51.649895 22266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4ce07a334a473c9935125ab10f4953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32911 } }
I20260806 04:43:51.650022 22266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:51.650063 22266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:51.650132 22266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:51.650983 22266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4ce07a334a473c9935125ab10f4953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32911 } }
I20260806 04:43:51.651330 22266 leader_election.cc:302] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c4ce07a334a473c9935125ab10f4953; no voters:
I20260806 04:43:51.651594 22266 leader_election.cc:288] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:51.651729 22278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:51.651980 22278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [term 1 LEADER]: Becoming Leader. State: Replica: 4c4ce07a334a473c9935125ab10f4953, State: Running, Role: LEADER
I20260806 04:43:51.652264 22278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4ce07a334a473c9935125ab10f4953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32911 } }
I20260806 04:43:51.652810 22266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:51.653891 22279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c4ce07a334a473c9935125ab10f4953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4ce07a334a473c9935125ab10f4953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32911 } } }
I20260806 04:43:51.654036 22279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:51.654390 22280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c4ce07a334a473c9935125ab10f4953. Latest consensus state: current_term: 1 leader_uuid: "4c4ce07a334a473c9935125ab10f4953" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c4ce07a334a473c9935125ab10f4953" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32911 } } }
I20260806 04:43:51.654482 22280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:51.654627 22287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:51.657583 22287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:51.663189 22287 catalog_manager.cc:1347] Generated new cluster ID: 9d316685e3754961be2146482b2bbf98
I20260806 04:43:51.663276 22287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:51.703987 22287 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:51.704335 22287 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:51.709383 22287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c4ce07a334a473c9935125ab10f4953: Generated new TSK 0
I20260806 04:43:51.710075 22287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:43:51.716955 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 22195
2026-08-06T04:43:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1465 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:51 dist-test-slave-vkjp krb5kdc[22299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:51 dist-test-slave-vkjp krb5kdc[22299](info): set up 2 sockets
Aug 06 04:43:51 dist-test-slave-vkjp krb5kdc[22299](info): commencing operation
krb5kdc: starting...
W20260806 04:43:52.797233 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:43:52 dist-test-slave-vkjp krb5kdc[22299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:43:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:43:52.927068 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:53.044507 22312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:53.044813 22312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:53.044893 22312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:53.049027 22312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:53.049120 22312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:53.049158 22312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:53.049219 22312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:53.049259 22312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:53.054672 22312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:53.055935 22312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:53.057067 22312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:53.066192 22318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:53.066181 22322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:53.066463 22312 server_base.cc:1034] running on GCE node
W20260806 04:43:53.066584 22319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:53.067277 22312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:53.068475 22312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:53.069674 22312 hybrid_clock.cc:648] HybridClock initialized: now 1785991433069659 us; error 32 us; skew 500 ppm
Aug 06 04:43:53 dist-test-slave-vkjp krb5kdc[22299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:53.072777 22312 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:43:53.073787 22312 webserver.cc:460] Webserver started at http://127.0.0.1:45437/ using document root <none> and password file <none>
I20260806 04:43:53.074540 22312 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:53.074674 22312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:53.074954 22312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:53.077471 22312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "f9a9608d1591482d8ffc754522d0c532"
format_stamp: "Formatted at 2026-08-06 04:43:53 on dist-test-slave-vkjp"
server_key: "f6f82fd9a8453cfa2954d4defc079b9b"
server_key_iv: "d301194e0e81ddc31c48061930a3d49c"
server_key_version: "encryptionkey@0"
I20260806 04:43:53.078181 22312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "f9a9608d1591482d8ffc754522d0c532"
format_stamp: "Formatted at 2026-08-06 04:43:53 on dist-test-slave-vkjp"
server_key: "f6f82fd9a8453cfa2954d4defc079b9b"
server_key_iv: "d301194e0e81ddc31c48061930a3d49c"
server_key_version: "encryptionkey@0"
I20260806 04:43:53.082926 22312 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:43:53.085901 22329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:53.087404 22312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:43:53.087560 22312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "f9a9608d1591482d8ffc754522d0c532"
format_stamp: "Formatted at 2026-08-06 04:43:53 on dist-test-slave-vkjp"
server_key: "f6f82fd9a8453cfa2954d4defc079b9b"
server_key_iv: "d301194e0e81ddc31c48061930a3d49c"
server_key_version: "encryptionkey@0"
I20260806 04:43:53.087690 22312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:53.109885 22312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:53.113720 22312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:53.113956 22312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:53.121882 22312 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36385
I20260806 04:43:53.121909 22385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36385 every 8 connection(s)
I20260806 04:43:53.122969 22312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:53.124047 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 22312
I20260806 04:43:53.124199 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:53.124533 32410 external_mini_cluster.cc:1428] Setting key dcd205f3826f16d0037efef4d62db1b1
I20260806 04:43:53.126260 22386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:53.131865 22386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532: Bootstrap starting.
Aug 06 04:43:53 dist-test-slave-vkjp krb5kdc[22299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 04:43:53.134385 22386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:53.135206 22386 log.cc:824] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:53.137843 22386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532: No bootstrap required, opened a new log
I20260806 04:43:53.140322 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:53.126011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:52611 (local address 127.0.0.1:36385)
0806 04:43:53.126539 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:53.126554 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:53.126592 (+ 38us) server_negotiation.cc:407] Connection header received
0806 04:43:53.127358 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:53.127377 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:53.127709 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:53.128044 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:53.128931 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:53.129919 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:53.130841 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:53.131386 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:53.131557 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:53.134576 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:53.134582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:53.134602 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:53.134645 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:53.136674 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:53.137408 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:53.137413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:53.137419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:53.137535 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:53.138065 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:53.138068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:53.138069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:53.138502 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:53.138665 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:53.139182 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 04:43:53.139499 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":172,"threads_started":1}
I20260806 04:43:53.142319 22386 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a9608d1591482d8ffc754522d0c532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260806 04:43:53.142573 22386 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:53.142663 22386 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a9608d1591482d8ffc754522d0c532, State: Initialized, Role: FOLLOWER
I20260806 04:43:53.143209 22386 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a9608d1591482d8ffc754522d0c532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260806 04:43:53.143410 22386 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:53.143522 22386 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:53.143668 22386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:53.143869 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:43:53.144917 22386 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a9608d1591482d8ffc754522d0c532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260806 04:43:53.145341 22386 leader_election.cc:302] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9a9608d1591482d8ffc754522d0c532; no voters:
I20260806 04:43:53.145744 22386 leader_election.cc:288] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:53.145879 22396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:53.146152 22396 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [term 1 LEADER]: Becoming Leader. State: Replica: f9a9608d1591482d8ffc754522d0c532, State: Running, Role: LEADER
I20260806 04:43:53.146554 22396 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a9608d1591482d8ffc754522d0c532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } }
I20260806 04:43:53.147333 22386 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:53.147912 22398 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9a9608d1591482d8ffc754522d0c532. Latest consensus state: current_term: 1 leader_uuid: "f9a9608d1591482d8ffc754522d0c532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a9608d1591482d8ffc754522d0c532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } } }
I20260806 04:43:53.148063 22398 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:53.148881 22397 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9a9608d1591482d8ffc754522d0c532" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9a9608d1591482d8ffc754522d0c532" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36385 } } }
I20260806 04:43:53.148999 22397 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:53.149796 22401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:53.153393 22401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:53.156875 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:53.146494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:56501 (local address 127.0.0.1:36385)
0806 04:43:53.146600 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:53.146616 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:53.146734 (+ 118us) server_negotiation.cc:407] Connection header received
0806 04:43:53.146941 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:53.146945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:53.147021 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:53.147127 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:53.148135 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:53.149009 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:53.150295 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:53.150642 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:53.150727 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:53.152748 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:53.152751 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:53.152753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:53.152791 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:53.154619 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:53.155298 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:53.155302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:53.155304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:53.155400 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:53.155841 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:53.155845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:53.155847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:53.156046 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:53.156151 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:53.156568 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 04:43:53.156675 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 04:43:53.161490 22401 catalog_manager.cc:1347] Generated new cluster ID: f88c6078c0194849923bfcc1c86d24f3
I20260806 04:43:53.161605 22401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:53.182051 22401 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:53.183250 22401 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:53.191897 22401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9a9608d1591482d8ffc754522d0c532: Generated new TSK 0
I20260806 04:43:53.192764 22401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:43:53.202651 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 22312
W20260806 04:43:53.209980 22297 connection.cc:535] client connection to 127.0.0.1:36385 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36385): Connection reset by peer (error 104)
2026-08-06T04:43:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1486 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:53 dist-test-slave-vkjp krb5kdc[22418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:53 dist-test-slave-vkjp krb5kdc[22418](info): set up 2 sockets
Aug 06 04:43:53 dist-test-slave-vkjp krb5kdc[22418](info): commencing operation
krb5kdc: starting...
W20260806 04:43:54.285919 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:43:54 dist-test-slave-vkjp krb5kdc[22418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:43:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:43:54.422976 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:54.534152 22431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:54.534409 22431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:54.534451 22431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:54.538468 22431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:54.538524 22431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:54.538543 22431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:54.538560 22431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:54.538575 22431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:54.543758 22431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:54.544716 22431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:54.545651 22431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:54.554194 22440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:54.554400 22431 server_base.cc:1034] running on GCE node
W20260806 04:43:54.554153 22438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:54.554165 22436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:54.555068 22431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:54.556232 22431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:54.557421 22431 hybrid_clock.cc:648] HybridClock initialized: now 1785991434557408 us; error 26 us; skew 500 ppm
Aug 06 04:43:54 dist-test-slave-vkjp krb5kdc[22418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:54.560370 22431 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:43:54.561383 22431 webserver.cc:460] Webserver started at http://127.0.0.1:43769/ using document root <none> and password file <none>
I20260806 04:43:54.561939 22431 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:54.562028 22431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:54.562261 22431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:54.564352 22431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "3d6383f7e32340c4b01440dd92b7554f"
format_stamp: "Formatted at 2026-08-06 04:43:54 on dist-test-slave-vkjp"
server_key: "3d6f4952d5baea6e2f9b0f054b18ea93"
server_key_iv: "662f7f412fa122dc194f11f2a953ba0c"
server_key_version: "encryptionkey@0"
I20260806 04:43:54.564911 22431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "3d6383f7e32340c4b01440dd92b7554f"
format_stamp: "Formatted at 2026-08-06 04:43:54 on dist-test-slave-vkjp"
server_key: "3d6f4952d5baea6e2f9b0f054b18ea93"
server_key_iv: "662f7f412fa122dc194f11f2a953ba0c"
server_key_version: "encryptionkey@0"
I20260806 04:43:54.568745 22431 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 04:43:54.571274 22447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:54.572518 22431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:43:54.572641 22431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "3d6383f7e32340c4b01440dd92b7554f"
format_stamp: "Formatted at 2026-08-06 04:43:54 on dist-test-slave-vkjp"
server_key: "3d6f4952d5baea6e2f9b0f054b18ea93"
server_key_iv: "662f7f412fa122dc194f11f2a953ba0c"
server_key_version: "encryptionkey@0"
I20260806 04:43:54.572744 22431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:54.583886 22431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:54.587200 22431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:54.587396 22431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:54.594821 22431 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39561
I20260806 04:43:54.594960 22503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39561 every 8 connection(s)
I20260806 04:43:54.595893 22431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:54.598856 22504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:54.599377 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 22431
I20260806 04:43:54.599525 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:54.599835 32410 external_mini_cluster.cc:1428] Setting key 17456378ff90c04405b1252f6132c0b9
I20260806 04:43:54.606397 22504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f: Bootstrap starting.
I20260806 04:43:54.609022 22504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:54.609400 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:54.601417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:39099 (local address 127.0.0.1:39561)
0806 04:43:54.601904 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:54.601918 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:54.601953 (+ 35us) server_negotiation.cc:407] Connection header received
0806 04:43:54.602740 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:54.602759 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:54.603176 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:54.603659 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:54.604084 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:54.605058 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:54.605973 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:54.606443 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:54.606614 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:54.607653 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:54.607658 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:54.607670 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:54.608065 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:54.608123 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:54.608145 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:54.608292 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:54.608488 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 04:43:54.608636 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":158,"threads_started":1}
I20260806 04:43:54.610009 22504 log.cc:824] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:54.612282 22504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f: No bootstrap required, opened a new log
I20260806 04:43:54.612854 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:43:54.616348 22504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6383f7e32340c4b01440dd92b7554f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } }
I20260806 04:43:54.616645 22504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:54.616707 22504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6383f7e32340c4b01440dd92b7554f, State: Initialized, Role: FOLLOWER
I20260806 04:43:54.617166 22504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6383f7e32340c4b01440dd92b7554f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } }
I20260806 04:43:54.617316 22504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:54.617384 22504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:54.617503 22504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:54.618692 22504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6383f7e32340c4b01440dd92b7554f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } }
I20260806 04:43:54.619133 22504 leader_election.cc:302] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d6383f7e32340c4b01440dd92b7554f; no voters:
I20260806 04:43:54.619525 22504 leader_election.cc:288] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:54.619578 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:54.615027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:52457 (local address 127.0.0.1:39561)
0806 04:43:54.615177 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:54.615181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:54.615327 (+ 146us) server_negotiation.cc:407] Connection header received
0806 04:43:54.615425 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:54.615427 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:54.615468 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:54.615535 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:54.615836 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:54.616361 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:54.617483 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:54.617824 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:54.617908 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:54.618891 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:54.618895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:54.618898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:54.618914 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:54.618953 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:54.618968 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:54.619059 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:54.619348 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 04:43:54.619393 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 04:43:54.619619 22514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:54.619917 22514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [term 1 LEADER]: Becoming Leader. State: Replica: 3d6383f7e32340c4b01440dd92b7554f, State: Running, Role: LEADER
I20260806 04:43:54.620357 22514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6383f7e32340c4b01440dd92b7554f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } }
I20260806 04:43:54.620867 22504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:54.621706 22515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d6383f7e32340c4b01440dd92b7554f. Latest consensus state: current_term: 1 leader_uuid: "3d6383f7e32340c4b01440dd92b7554f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6383f7e32340c4b01440dd92b7554f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } } }
I20260806 04:43:54.621846 22515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:54.622216 22517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d6383f7e32340c4b01440dd92b7554f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6383f7e32340c4b01440dd92b7554f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } } }
I20260806 04:43:54.622421 22517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:54.622763 22524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:54.626371 22524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:54.632596 22524 catalog_manager.cc:1347] Generated new cluster ID: 3bc87ab1b87044bb8576dc4a4c5fafe9
I20260806 04:43:54.632748 22524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:54.650851 22524 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:54.652171 22524 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:54.662016 22524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d6383f7e32340c4b01440dd92b7554f: Generated new TSK 0
I20260806 04:43:54.662729 22524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:43:54.671680 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 22431
W20260806 04:43:54.678400 22416 connection.cc:535] client connection to 127.0.0.1:39561 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39561): Connection reset by peer (error 104)
2026-08-06T04:43:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1468 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:54 dist-test-slave-vkjp krb5kdc[22536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:54 dist-test-slave-vkjp krb5kdc[22536](info): set up 2 sockets
Aug 06 04:43:54 dist-test-slave-vkjp krb5kdc[22536](info): commencing operation
krb5kdc: starting...
W20260806 04:43:55.837333 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:43:55 dist-test-slave-vkjp krb5kdc[22536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:43:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:43:55.962706 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:56.086644 22550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:56.086926 22550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:56.086972 22550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:56.090898 22550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:56.090962 22550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:56.090986 22550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:56.091005 22550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:56.091023 22550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:56.096415 22550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:56.097443 22550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:56.098559 22550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:56.107278 22560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:56.107566 22550 server_base.cc:1034] running on GCE node
W20260806 04:43:56.107300 22558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:56.107369 22556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:56.108249 22550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:56.109318 22550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:56.110539 22550 hybrid_clock.cc:648] HybridClock initialized: now 1785991436110514 us; error 44 us; skew 500 ppm
Aug 06 04:43:56 dist-test-slave-vkjp krb5kdc[22536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:56.113333 22550 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:43:56.114202 22550 webserver.cc:460] Webserver started at http://127.0.0.1:36377/ using document root <none> and password file <none>
I20260806 04:43:56.114720 22550 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:56.114782 22550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:56.114945 22550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:56.117009 22550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "d7dfdc363b6b4f9a93f112009d19c6ed"
format_stamp: "Formatted at 2026-08-06 04:43:56 on dist-test-slave-vkjp"
server_key: "c42a1676462094409a50d3854472ae99"
server_key_iv: "8b060d78277bbe2bf3db9c65affaac98"
server_key_version: "encryptionkey@0"
I20260806 04:43:56.117516 22550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "d7dfdc363b6b4f9a93f112009d19c6ed"
format_stamp: "Formatted at 2026-08-06 04:43:56 on dist-test-slave-vkjp"
server_key: "c42a1676462094409a50d3854472ae99"
server_key_iv: "8b060d78277bbe2bf3db9c65affaac98"
server_key_version: "encryptionkey@0"
I20260806 04:43:56.121198 22550 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:43:56.123762 22567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:56.125001 22550 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260806 04:43:56.125105 22550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "d7dfdc363b6b4f9a93f112009d19c6ed"
format_stamp: "Formatted at 2026-08-06 04:43:56 on dist-test-slave-vkjp"
server_key: "c42a1676462094409a50d3854472ae99"
server_key_iv: "8b060d78277bbe2bf3db9c65affaac98"
server_key_version: "encryptionkey@0"
I20260806 04:43:56.125195 22550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:56.129784 22550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:56.132756 22550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:56.132902 22550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:56.140192 22550 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46049
I20260806 04:43:56.140216 22621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46049 every 8 connection(s)
I20260806 04:43:56.141115 22550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:56.143937 22622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:56.149839 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 22550
I20260806 04:43:56.149953 22622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed: Bootstrap starting.
I20260806 04:43:56.149994 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:56.150326 32410 external_mini_cluster.cc:1428] Setting key ee003c5c6c0abe6ab07af9af6e5884b3
I20260806 04:43:56.152714 22622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:56.153450 22622 log.cc:824] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:56.155560 22622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed: No bootstrap required, opened a new log
I20260806 04:43:56.155814 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:56.151790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:49229 (local address 127.0.0.1:46049)
0806 04:43:56.152245 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:56.152260 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:56.152296 (+ 36us) server_negotiation.cc:407] Connection header received
0806 04:43:56.153010 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:56.153029 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:56.153317 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:56.153648 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:56.154505 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:56.154509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:56.154520 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:56.154531 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:56.154584 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:56.154603 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:56.154722 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:56.154826 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 04:43:56.154996 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":138,"threads_started":1}
I20260806 04:43:56.158762 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:43:56.159837 22622 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dfdc363b6b4f9a93f112009d19c6ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46049 } }
I20260806 04:43:56.160149 22622 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:56.160223 22622 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7dfdc363b6b4f9a93f112009d19c6ed, State: Initialized, Role: FOLLOWER
I20260806 04:43:56.160629 22622 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dfdc363b6b4f9a93f112009d19c6ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46049 } }
I20260806 04:43:56.160817 22622 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:56.160909 22622 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:56.161038 22622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:56.161988 22622 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dfdc363b6b4f9a93f112009d19c6ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46049 } }
I20260806 04:43:56.162318 22622 leader_election.cc:302] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7dfdc363b6b4f9a93f112009d19c6ed; no voters:
I20260806 04:43:56.162645 22622 leader_election.cc:288] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:56.162707 22632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:56.162887 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:56.161005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:58013 (local address 127.0.0.1:46049)
0806 04:43:56.161136 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:56.161143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:56.161406 (+ 263us) server_negotiation.cc:407] Connection header received
0806 04:43:56.161552 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:56.161555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:56.161595 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:56.161677 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:56.162435 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:56.162439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:56.162444 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:56.162458 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:56.162486 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:56.162497 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:56.162594 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:56.162690 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 04:43:56.162734 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 04:43:56.162964 22632 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [term 1 LEADER]: Becoming Leader. State: Replica: d7dfdc363b6b4f9a93f112009d19c6ed, State: Running, Role: LEADER
I20260806 04:43:56.163280 22632 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dfdc363b6b4f9a93f112009d19c6ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46049 } }
I20260806 04:43:56.164026 22622 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:56.164462 22632 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7dfdc363b6b4f9a93f112009d19c6ed. Latest consensus state: current_term: 1 leader_uuid: "d7dfdc363b6b4f9a93f112009d19c6ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dfdc363b6b4f9a93f112009d19c6ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46049 } } }
I20260806 04:43:56.164580 22632 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:56.164780 22632 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7dfdc363b6b4f9a93f112009d19c6ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dfdc363b6b4f9a93f112009d19c6ed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46049 } } }
I20260806 04:43:56.164955 22632 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:56.164961 22637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:56.168391 22637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:56.174280 22637 catalog_manager.cc:1347] Generated new cluster ID: 25130956ce44443a809ef12a2e8b8136
I20260806 04:43:56.174367 22637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:56.184000 22637 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:56.184310 22637 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:56.189746 22637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7dfdc363b6b4f9a93f112009d19c6ed: Generated new TSK 0
I20260806 04:43:56.190460 22637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:43:56.199208 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 22550
2026-08-06T04:43:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1528 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:56 dist-test-slave-vkjp krb5kdc[22655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:56 dist-test-slave-vkjp krb5kdc[22655](info): set up 2 sockets
Aug 06 04:43:56 dist-test-slave-vkjp krb5kdc[22655](info): commencing operation
krb5kdc: starting...
W20260806 04:43:57.282522 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:43:57 dist-test-slave-vkjp krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:43:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:43:57.414991 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:57.527506 22669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:57.527771 22669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:57.527814 22669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:57.531852 22669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:57.531917 22669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:57.531937 22669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:57.531953 22669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:57.531989 22669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:57.537240 22669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39789
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:57.538429 22669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:57.539541 22669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:57.547885 22675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:57.548022 22677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:57.547919 22669 server_base.cc:1034] running on GCE node
W20260806 04:43:57.547936 22674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:57.548794 22669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:57.549811 22669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:57.550997 22669 hybrid_clock.cc:648] HybridClock initialized: now 1785991437550978 us; error 30 us; skew 500 ppm
Aug 06 04:43:57 dist-test-slave-vkjp krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:57.559650 22669 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:43:57.560699 22669 webserver.cc:460] Webserver started at http://127.0.0.1:38963/ using document root <none> and password file <none>
I20260806 04:43:57.561304 22669 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:57.561369 22669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:57.561616 22669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:57.563807 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "f76e2cb0514c4808a6160c7edd3545e0"
format_stamp: "Formatted at 2026-08-06 04:43:57 on dist-test-slave-vkjp"
server_key: "67580506bd1143130fb0f2d5f85a2465"
server_key_iv: "33e5569bffef0ed68dc988e7e387252b"
server_key_version: "encryptionkey@0"
I20260806 04:43:57.564369 22669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "f76e2cb0514c4808a6160c7edd3545e0"
format_stamp: "Formatted at 2026-08-06 04:43:57 on dist-test-slave-vkjp"
server_key: "67580506bd1143130fb0f2d5f85a2465"
server_key_iv: "33e5569bffef0ed68dc988e7e387252b"
server_key_version: "encryptionkey@0"
I20260806 04:43:57.568183 22669 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 04:43:57.570606 22685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:57.572073 22669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:43:57.572223 22669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "f76e2cb0514c4808a6160c7edd3545e0"
format_stamp: "Formatted at 2026-08-06 04:43:57 on dist-test-slave-vkjp"
server_key: "67580506bd1143130fb0f2d5f85a2465"
server_key_iv: "33e5569bffef0ed68dc988e7e387252b"
server_key_version: "encryptionkey@0"
I20260806 04:43:57.572346 22669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:57.590854 22669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:57.594740 22669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:57.594974 22669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:57.602787 22669 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39789
I20260806 04:43:57.602802 22738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39789 every 8 connection(s)
I20260806 04:43:57.603890 22669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:57.606770 22739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:57.612116 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 22669
I20260806 04:43:57.612232 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:57.612541 32410 external_mini_cluster.cc:1428] Setting key 4d722f2c973b6939259ad8ffd2700e4f
I20260806 04:43:57.612828 22739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0: Bootstrap starting.
I20260806 04:43:57.615809 22739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:57.616761 22739 log.cc:824] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:57.619204 22739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0: No bootstrap required, opened a new log
Aug 06 04:43:57 dist-test-slave-vkjp krb5kdc[22655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 04:43:57.623849 22739 raft_consensus.cc:348] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76e2cb0514c4808a6160c7edd3545e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39789 } }
I20260806 04:43:57.624164 22739 raft_consensus.cc:374] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:57.624315 22739 raft_consensus.cc:729] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76e2cb0514c4808a6160c7edd3545e0, State: Initialized, Role: FOLLOWER
I20260806 04:43:57.624800 22739 consensus_queue.cc:260] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76e2cb0514c4808a6160c7edd3545e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39789 } }
I20260806 04:43:57.624961 22739 raft_consensus.cc:388] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:57.625039 22739 raft_consensus.cc:482] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:57.625173 22739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:57.626259 22739 raft_consensus.cc:504] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76e2cb0514c4808a6160c7edd3545e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39789 } }
I20260806 04:43:57.626650 22739 leader_election.cc:302] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f76e2cb0514c4808a6160c7edd3545e0; no voters:
I20260806 04:43:57.627027 22739 leader_election.cc:288] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:57.627130 22744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:57.627410 22744 raft_consensus.cc:686] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [term 1 LEADER]: Becoming Leader. State: Replica: f76e2cb0514c4808a6160c7edd3545e0, State: Running, Role: LEADER
I20260806 04:43:57.627894 22744 consensus_queue.cc:237] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76e2cb0514c4808a6160c7edd3545e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39789 } }
I20260806 04:43:57.628526 22739 sys_catalog.cc:564] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:57.629559 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:57.614107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:48327 (local address 127.0.0.1:39789)
0806 04:43:57.614569 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:57.614590 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:57.614638 (+ 48us) server_negotiation.cc:407] Connection header received
0806 04:43:57.615575 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:57.615601 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:57.616084 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:57.616472 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:57.617335 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:57.618754 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:57.619805 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:57.620486 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:57.620660 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:57.623650 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:57.623658 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:57.623675 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:57.623712 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:57.625873 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:57.626579 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:57.626585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:57.626589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:57.626712 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:57.627229 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:57.627234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:57.627235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:57.627772 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:57.627995 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:57.628374 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 04:43:57.628647 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":135,"threads_started":1}
I20260806 04:43:57.629719 22746 sys_catalog.cc:455] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f76e2cb0514c4808a6160c7edd3545e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76e2cb0514c4808a6160c7edd3545e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39789 } } }
I20260806 04:43:57.629889 22746 sys_catalog.cc:458] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:57.629956 22745 sys_catalog.cc:455] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f76e2cb0514c4808a6160c7edd3545e0. Latest consensus state: current_term: 1 leader_uuid: "f76e2cb0514c4808a6160c7edd3545e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f76e2cb0514c4808a6160c7edd3545e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39789 } } }
I20260806 04:43:57.630035 22745 sys_catalog.cc:458] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:57.630323 22753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:57.633697 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:43:57.633859 22753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:57.640733 22753 catalog_manager.cc:1347] Generated new cluster ID: 8fbf0eafdf534d64a09d40349235f2b6
I20260806 04:43:57.640836 22753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:57.646688 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:57.636647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:34819 (local address 127.0.0.1:39789)
0806 04:43:57.636784 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:57.636789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:57.637045 (+ 256us) server_negotiation.cc:407] Connection header received
0806 04:43:57.637216 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:57.637220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:57.637280 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:57.637392 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:57.638148 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:57.639420 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:57.640328 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:57.640696 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:57.640826 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:57.642829 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:57.642833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:57.642836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:43:57.642865 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:57.644684 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:57.645254 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:57.645258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:57.645260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:57.645350 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:43:57.645705 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:43:57.645709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:43:57.645711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:43:57.645954 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:57.646107 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:57.646415 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 04:43:57.646521 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:43:57.658245 22753 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:57.659574 22753 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:57.664604 22753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f76e2cb0514c4808a6160c7edd3545e0: Generated new TSK 0
I20260806 04:43:57.665284 22753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:43:57.669502 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 22669
W20260806 04:43:57.677124 22653 connection.cc:535] client connection to 127.0.0.1:39789 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39789): Connection reset by peer (error 104)
2026-08-06T04:43:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1469 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:57 dist-test-slave-vkjp krb5kdc[22771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:57 dist-test-slave-vkjp krb5kdc[22771](info): set up 2 sockets
Aug 06 04:43:57 dist-test-slave-vkjp krb5kdc[22771](info): commencing operation
krb5kdc: starting...
W20260806 04:43:58.750522 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:43:58 dist-test-slave-vkjp krb5kdc[22771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:43:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:43:58.886162 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35521
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:43:58.999511 22785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:43:58.999863 22785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:43:58.999946 22785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:43:59.004668 22785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:43:59.004776 22785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:43:59.004818 22785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:43:59.004855 22785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:43:59.004889 22785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:43:59.010529 22785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35521
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:43:59.011821 22785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:43:59.012878 22785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:43:59.023520 22793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:59.023554 22785 server_base.cc:1034] running on GCE node
W20260806 04:43:59.023543 22792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:43:59.023550 22795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:43:59.024513 22785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:43:59.026026 22785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:43:59.027230 22785 hybrid_clock.cc:648] HybridClock initialized: now 1785991439027221 us; error 32 us; skew 500 ppm
Aug 06 04:43:59 dist-test-slave-vkjp krb5kdc[22771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:43:59.030095 22785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:43:59.031031 22785 webserver.cc:460] Webserver started at http://127.0.0.1:45235/ using document root <none> and password file <none>
I20260806 04:43:59.031656 22785 fs_manager.cc:359] Metadata directory not provided
I20260806 04:43:59.031754 22785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:43:59.032007 22785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:43:59.034121 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "d631b56c1b26472e84722809a10d7347"
format_stamp: "Formatted at 2026-08-06 04:43:59 on dist-test-slave-vkjp"
server_key: "d86cb35b4ef42e3e3342985d2c3bde54"
server_key_iv: "438eb1babad95fd1b24b10374f01138a"
server_key_version: "encryptionkey@0"
I20260806 04:43:59.034708 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "d631b56c1b26472e84722809a10d7347"
format_stamp: "Formatted at 2026-08-06 04:43:59 on dist-test-slave-vkjp"
server_key: "d86cb35b4ef42e3e3342985d2c3bde54"
server_key_iv: "438eb1babad95fd1b24b10374f01138a"
server_key_version: "encryptionkey@0"
I20260806 04:43:59.038564 22785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:43:59.041272 22802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:43:59.042609 22785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 04:43:59.042770 22785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "d631b56c1b26472e84722809a10d7347"
format_stamp: "Formatted at 2026-08-06 04:43:59 on dist-test-slave-vkjp"
server_key: "d86cb35b4ef42e3e3342985d2c3bde54"
server_key_iv: "438eb1babad95fd1b24b10374f01138a"
server_key_version: "encryptionkey@0"
I20260806 04:43:59.042912 22785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:43:59.055289 22785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:43:59.058627 22785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:43:59.058861 22785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:43:59.066380 22785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35521
I20260806 04:43:59.066421 22857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35521 every 8 connection(s)
I20260806 04:43:59.067402 22785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:43:59.071839 22858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:43:59.073176 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 22785
I20260806 04:43:59.073330 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:43:59.073671 32410 external_mini_cluster.cc:1428] Setting key f246997164de04141968b2770611f47e
I20260806 04:43:59.080466 22858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347: Bootstrap starting.
I20260806 04:43:59.083806 22858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347: Neither blocks nor log segments found. Creating new log.
I20260806 04:43:59.084849 22858 log.cc:824] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347: Log is configured to *not* fsync() on all Append() calls
I20260806 04:43:59.085963 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:59.075230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:57825 (local address 127.0.0.1:35521)
0806 04:43:59.075796 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:59.075816 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:59.075881 (+ 65us) server_negotiation.cc:407] Connection header received
0806 04:43:59.076929 (+ 1048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:59.076954 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:59.077434 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:59.077826 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:59.078234 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:59.079698 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:59.081171 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:59.081980 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:59.082187 (+ 207us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:59.083368 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:59.083376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:59.083394 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:59.084022 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:59.084119 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:59.084164 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:59.084335 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:59.084591 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 04:43:59.084754 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":149,"threads_started":1}
I20260806 04:43:59.087560 22858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347: No bootstrap required, opened a new log
I20260806 04:43:59.089896 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:43:59.093083 22858 raft_consensus.cc:348] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d631b56c1b26472e84722809a10d7347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } }
I20260806 04:43:59.093410 22858 raft_consensus.cc:374] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:43:59.093503 22858 raft_consensus.cc:729] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d631b56c1b26472e84722809a10d7347, State: Initialized, Role: FOLLOWER
I20260806 04:43:59.094048 22858 consensus_queue.cc:260] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d631b56c1b26472e84722809a10d7347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } }
I20260806 04:43:59.094218 22858 raft_consensus.cc:388] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:43:59.094334 22858 raft_consensus.cc:482] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:43:59.094449 22858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:43:59.095921 22858 raft_consensus.cc:504] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d631b56c1b26472e84722809a10d7347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } }
I20260806 04:43:59.096328 22858 leader_election.cc:302] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d631b56c1b26472e84722809a10d7347; no voters:
I20260806 04:43:59.096807 22858 leader_election.cc:288] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:43:59.097338 22868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:43:59.097463 22861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:43:59.092000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:42193 (local address 127.0.0.1:35521)
0806 04:43:59.092152 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 04:43:59.092156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:43:59.092405 (+ 249us) server_negotiation.cc:407] Connection header received
0806 04:43:59.092519 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:43:59.092522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:43:59.092594 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:43:59.092690 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:43:59.093019 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:59.093873 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:59.095014 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:43:59.095479 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:43:59.095595 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:43:59.096854 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:43:59.096860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:43:59.096863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:43:59.096878 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:43:59.096908 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:43:59.096922 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:43:59.097023 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 04:43:59.097227 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 04:43:59.097266 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 04:43:59.097615 22868 raft_consensus.cc:686] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [term 1 LEADER]: Becoming Leader. State: Replica: d631b56c1b26472e84722809a10d7347, State: Running, Role: LEADER
I20260806 04:43:59.097983 22868 consensus_queue.cc:237] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d631b56c1b26472e84722809a10d7347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } }
I20260806 04:43:59.098253 22858 sys_catalog.cc:564] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:43:59.099711 22871 sys_catalog.cc:455] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d631b56c1b26472e84722809a10d7347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d631b56c1b26472e84722809a10d7347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } } }
I20260806 04:43:59.099938 22871 sys_catalog.cc:458] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [sys.catalog]: This master's current role is: LEADER
I20260806 04:43:59.100338 22870 sys_catalog.cc:455] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d631b56c1b26472e84722809a10d7347. Latest consensus state: current_term: 1 leader_uuid: "d631b56c1b26472e84722809a10d7347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d631b56c1b26472e84722809a10d7347" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35521 } } }
I20260806 04:43:59.100442 22870 sys_catalog.cc:458] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347 [sys.catalog]: This master's current role is: LEADER
W20260806 04:43:59.103982 22884 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:43:59.104127 22884 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:43:59.104197 22881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:43:59.104882 22881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:43:59.111789 22881 catalog_manager.cc:1347] Generated new cluster ID: 9d78b6e8728c4ec298143972e679e182
I20260806 04:43:59.111865 22881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:43:59.119757 22881 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:43:59.120769 22881 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:43:59.130313 22881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d631b56c1b26472e84722809a10d7347: Generated new TSK 0
I20260806 04:43:59.131027 22881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:43:59.142975 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 22785
W20260806 04:43:59.149363 22768 connection.cc:535] client connection to 127.0.0.1:35521 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35521): Connection reset by peer (error 104)
2026-08-06T04:43:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1472 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:43:59 dist-test-slave-vkjp krb5kdc[22891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:43:59 dist-test-slave-vkjp krb5kdc[22891](info): set up 2 sockets
Aug 06 04:43:59 dist-test-slave-vkjp krb5kdc[22891](info): commencing operation
krb5kdc: starting...
W20260806 04:44:00.226119 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:44:00 dist-test-slave-vkjp krb5kdc[22891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:44:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:44:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 04:44:00.358006 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:44:00.470069 22905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:44:00.470369 22905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:44:00.470414 22905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:44:00.474419 22905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:44:00.474488 22905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:44:00.474507 22905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:44:00.474524 22905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:44:00.474539 22905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:44:00.479966 22905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36917
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:44:00.481045 22905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:44:00.482049 22905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:44:00.490552 22913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:00.490739 22905 server_base.cc:1034] running on GCE node
W20260806 04:44:00.490547 22910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:00.490576 22911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:00.491508 22905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:44:00.492552 22905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:44:00.493765 22905 hybrid_clock.cc:648] HybridClock initialized: now 1785991440493745 us; error 28 us; skew 500 ppm
Aug 06 04:44:00 dist-test-slave-vkjp krb5kdc[22891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:44:00.496778 22905 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 04:44:00.497685 22905 webserver.cc:460] Webserver started at http://127.0.0.1:37169/ using document root <none> and password file <none>
I20260806 04:44:00.498214 22905 fs_manager.cc:359] Metadata directory not provided
I20260806 04:44:00.498282 22905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:44:00.498449 22905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:44:00.500592 22905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "71c9cb12cc6144659e54d8544ac277b4"
format_stamp: "Formatted at 2026-08-06 04:44:00 on dist-test-slave-vkjp"
server_key: "946ad2df56335c83a0391bd7833ecaf2"
server_key_iv: "836d57140fcb91421bc18c770f5295c4"
server_key_version: "encryptionkey@0"
I20260806 04:44:00.501144 22905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "71c9cb12cc6144659e54d8544ac277b4"
format_stamp: "Formatted at 2026-08-06 04:44:00 on dist-test-slave-vkjp"
server_key: "946ad2df56335c83a0391bd7833ecaf2"
server_key_iv: "836d57140fcb91421bc18c770f5295c4"
server_key_version: "encryptionkey@0"
I20260806 04:44:00.505142 22905 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:44:00.507666 22921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:44:00.508998 22905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:44:00.509104 22905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "71c9cb12cc6144659e54d8544ac277b4"
format_stamp: "Formatted at 2026-08-06 04:44:00 on dist-test-slave-vkjp"
server_key: "946ad2df56335c83a0391bd7833ecaf2"
server_key_iv: "836d57140fcb91421bc18c770f5295c4"
server_key_version: "encryptionkey@0"
I20260806 04:44:00.509248 22905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:44:00.514029 22905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:44:00.517300 22905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:44:00.517509 22905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:44:00.524242 22905 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36917
I20260806 04:44:00.524326 22974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36917 every 8 connection(s)
I20260806 04:44:00.525259 22905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:44:00.528211 22976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:44:00.533794 22976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4: Bootstrap starting.
I20260806 04:44:00.534689 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 22905
I20260806 04:44:00.534827 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:44:00.535210 32410 external_mini_cluster.cc:1428] Setting key be40f8f57c1976a98a1331fda914e0d8
I20260806 04:44:00.536855 22976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4: Neither blocks nor log segments found. Creating new log.
I20260806 04:44:00.537603 22976 log.cc:824] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4: Log is configured to *not* fsync() on all Append() calls
I20260806 04:44:00.539649 22976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4: No bootstrap required, opened a new log
I20260806 04:44:00.540851 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:00.536461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:55199 (local address 127.0.0.1:36917)
0806 04:44:00.536957 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:00.536999 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:00.537052 (+ 53us) server_negotiation.cc:407] Connection header received
0806 04:44:00.537884 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:00.537904 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:00.538199 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:00.538583 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:00.539480 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:00.539485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:00.539497 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:44:00.539509 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:00.539567 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:44:00.539588 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:00.539723 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:00.539847 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 04:44:00.540032 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":158,"threads_started":1}
I20260806 04:44:00.544046 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:44:00.544023 22976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c9cb12cc6144659e54d8544ac277b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36917 } }
I20260806 04:44:00.544312 22976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:44:00.544400 22976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71c9cb12cc6144659e54d8544ac277b4, State: Initialized, Role: FOLLOWER
I20260806 04:44:00.544807 22976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c9cb12cc6144659e54d8544ac277b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36917 } }
I20260806 04:44:00.544965 22976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:44:00.545051 22976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:44:00.545149 22976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:44:00.546347 22976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c9cb12cc6144659e54d8544ac277b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36917 } }
I20260806 04:44:00.546782 22976 leader_election.cc:302] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71c9cb12cc6144659e54d8544ac277b4; no voters:
I20260806 04:44:00.547183 22976 leader_election.cc:288] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:44:00.547245 22988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:44:00.547748 22988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [term 1 LEADER]: Becoming Leader. State: Replica: 71c9cb12cc6144659e54d8544ac277b4, State: Running, Role: LEADER
I20260806 04:44:00.548115 22988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c9cb12cc6144659e54d8544ac277b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36917 } }
I20260806 04:44:00.548316 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:00.546399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:57017 (local address 127.0.0.1:36917)
0806 04:44:00.546521 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:00.546526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:00.546741 (+ 215us) server_negotiation.cc:407] Connection header received
0806 04:44:00.546920 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:00.546923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:00.546963 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:00.547033 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:00.547867 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:00.547871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:00.547873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:44:00.547884 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:00.547912 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:44:00.547925 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:00.547991 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:00.548144 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 04:44:00.548189 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:44:00.548575 22976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:44:00.550120 22989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71c9cb12cc6144659e54d8544ac277b4. Latest consensus state: current_term: 1 leader_uuid: "71c9cb12cc6144659e54d8544ac277b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c9cb12cc6144659e54d8544ac277b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36917 } } }
I20260806 04:44:00.550148 22990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71c9cb12cc6144659e54d8544ac277b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71c9cb12cc6144659e54d8544ac277b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36917 } } }
I20260806 04:44:00.550266 22990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:00.550266 22989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4 [sys.catalog]: This master's current role is: LEADER
W20260806 04:44:00.553126 23005 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:44:00.553220 23005 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:44:00.553289 23002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:44:00.554028 23002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:44:00.560405 23002 catalog_manager.cc:1347] Generated new cluster ID: 3adedddc5e82407886e0bcde6f381820
I20260806 04:44:00.560487 23002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:44:00.573240 23002 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:44:00.573609 23002 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:44:00.580695 23002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71c9cb12cc6144659e54d8544ac277b4: Generated new TSK 0
I20260806 04:44:00.581458 23002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:44:00.594509 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 22905
2026-08-06T04:44:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:44:00 dist-test-slave-vkjp krb5kdc[23011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:44:00 dist-test-slave-vkjp krb5kdc[23011](info): set up 2 sockets
Aug 06 04:44:00 dist-test-slave-vkjp krb5kdc[23011](info): commencing operation
krb5kdc: starting...
W20260806 04:44:01.670209 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:44:01 dist-test-slave-vkjp krb5kdc[23011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:44:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:44:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:44:01.798934 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:34211
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:39175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:34211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:44:01.923286 23024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:44:01.923655 23024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:44:01.923753 23024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:44:01.928108 23024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:44:01.928217 23024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:44:01.928256 23024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:44:01.928323 23024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:44:01.928375 23024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:44:01.934049 23024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:39175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:34211
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:34211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:44:01.935335 23024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:44:01.936436 23024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:44:01.945915 23032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:01.945904 23030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:01.945904 23029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:01.945976 23024 server_base.cc:1034] running on GCE node
I20260806 04:44:01.946789 23024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:44:01.948191 23024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:44:01.949399 23024 hybrid_clock.cc:648] HybridClock initialized: now 1785991441949391 us; error 34 us; skew 500 ppm
Aug 06 04:44:01 dist-test-slave-vkjp krb5kdc[23011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991441, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:44:01.952795 23024 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:44:01.953835 23024 webserver.cc:460] Webserver started at http://127.31.166.190:43279/ using document root <none> and password file <none>
I20260806 04:44:01.954439 23024 fs_manager.cc:359] Metadata directory not provided
I20260806 04:44:01.954537 23024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:44:01.954790 23024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:44:01.957007 23024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "c5f904a549f04c608e6c577da6376eab"
format_stamp: "Formatted at 2026-08-06 04:44:01 on dist-test-slave-vkjp"
server_key: "6c00246ed43d80392f39d4fe001c6c2a"
server_key_iv: "0739d4a3015243094f15f0a641ddfc65"
server_key_version: "encryptionkey@0"
I20260806 04:44:01.957640 23024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "c5f904a549f04c608e6c577da6376eab"
format_stamp: "Formatted at 2026-08-06 04:44:01 on dist-test-slave-vkjp"
server_key: "6c00246ed43d80392f39d4fe001c6c2a"
server_key_iv: "0739d4a3015243094f15f0a641ddfc65"
server_key_version: "encryptionkey@0"
I20260806 04:44:01.961750 23024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:44:01.964643 23039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:44:01.965947 23024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 04:44:01.966105 23024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "c5f904a549f04c608e6c577da6376eab"
format_stamp: "Formatted at 2026-08-06 04:44:01 on dist-test-slave-vkjp"
server_key: "6c00246ed43d80392f39d4fe001c6c2a"
server_key_iv: "0739d4a3015243094f15f0a641ddfc65"
server_key_version: "encryptionkey@0"
I20260806 04:44:01.966238 23024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:44:01.987828 23024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:44:01.991341 23024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:44:01.991586 23024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:44:01.999481 23024 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:34211
I20260806 04:44:01.999521 23093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:34211 every 8 connection(s)
I20260806 04:44:02.000535 23024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:44:02.003752 23094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:44:02.006512 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 23024
I20260806 04:44:02.006888 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:44:02.007282 32410 external_mini_cluster.cc:1428] Setting key 462a0e44fe17aa130513fed42a364600
I20260806 04:44:02.010665 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab: Bootstrap starting.
I20260806 04:44:02.013592 23094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab: Neither blocks nor log segments found. Creating new log.
I20260806 04:44:02.014537 23094 log.cc:824] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab: Log is configured to *not* fsync() on all Append() calls
I20260806 04:44:02.016922 23094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab: No bootstrap required, opened a new log
Aug 06 04:44:02 dist-test-slave-vkjp krb5kdc[23011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:44:02.020911 23094 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f904a549f04c608e6c577da6376eab" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 34211 } }
I20260806 04:44:02.021175 23094 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:44:02.021246 23094 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5f904a549f04c608e6c577da6376eab, State: Initialized, Role: FOLLOWER
I20260806 04:44:02.021818 23094 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f904a549f04c608e6c577da6376eab" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 34211 } }
I20260806 04:44:02.021996 23094 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:44:02.022074 23094 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:44:02.022177 23094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:44:02.023548 23094 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f904a549f04c608e6c577da6376eab" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 34211 } }
I20260806 04:44:02.023967 23094 leader_election.cc:302] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5f904a549f04c608e6c577da6376eab; no voters:
I20260806 04:44:02.024312 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:02.008809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48743 (local address 127.31.166.190:34211)
0806 04:44:02.009235 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:02.009250 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:02.009288 (+ 38us) server_negotiation.cc:407] Connection header received
0806 04:44:02.009927 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:02.009944 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:02.010320 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:02.010596 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:02.011681 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:02.012741 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:02.014091 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:02.014595 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:02.014809 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:02.017849 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:02.017855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:02.017869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:02.017899 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:02.020503 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:02.021086 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:02.021091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:02.021096 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:02.021271 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:02.021717 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:02.021722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:02.021724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:02.022333 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:02.022589 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:02.022863 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 04:44:02.023146 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":141,"threads_started":1}
I20260806 04:44:02.024359 23094 leader_election.cc:288] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:44:02.024521 23099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:44:02.024772 23099 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [term 1 LEADER]: Becoming Leader. State: Replica: c5f904a549f04c608e6c577da6376eab, State: Running, Role: LEADER
I20260806 04:44:02.025125 23099 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f904a549f04c608e6c577da6376eab" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 34211 } }
I20260806 04:44:02.025717 23094 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:44:02.026373 23099 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5f904a549f04c608e6c577da6376eab. Latest consensus state: current_term: 1 leader_uuid: "c5f904a549f04c608e6c577da6376eab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f904a549f04c608e6c577da6376eab" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 34211 } } }
I20260806 04:44:02.026552 23099 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:02.027014 23106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:44:02.027789 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:44:02.029158 23100 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5f904a549f04c608e6c577da6376eab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5f904a549f04c608e6c577da6376eab" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 34211 } } }
I20260806 04:44:02.029302 23100 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:02.030880 23106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:44:02.039583 23106 catalog_manager.cc:1347] Generated new cluster ID: 544d9590ad1440f9a3e17351efcc5b47
I20260806 04:44:02.039649 23106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:44:02.040493 23097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:02.030002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50773 (local address 127.31.166.190:34211)
0806 04:44:02.030111 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:02.030116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:02.030645 (+ 529us) server_negotiation.cc:407] Connection header received
0806 04:44:02.030958 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:02.030997 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:02.031096 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:02.031192 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:02.032521 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:02.033393 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:02.034289 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:02.034632 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:02.034738 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:02.036623 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:02.036629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:02.036632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:02.036657 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:02.038153 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:02.038759 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:02.038763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:02.038764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:02.038856 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:02.039346 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:02.039350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:02.039352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:02.039593 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:02.039721 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:02.040224 (+ 503us) server_negotiation.cc:299] Negotiation successful
0806 04:44:02.040328 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 04:44:02.074497 23106 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:44:02.075505 23106 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:44:02.082896 23106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5f904a549f04c608e6c577da6376eab: Generated new TSK 0
I20260806 04:44:02.083699 23106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:44:02.097450 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 23024
W20260806 04:44:02.103845 23009 connection.cc:535] client connection to 127.31.166.190:34211 recv error: Network error: failed to read from TLS socket (remote: 127.31.166.190:34211): Connection reset by peer (error 104)
2026-08-06T04:44:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1502 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:44:02 dist-test-slave-vkjp krb5kdc[23127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:44:02 dist-test-slave-vkjp krb5kdc[23127](info): set up 2 sockets
Aug 06 04:44:02 dist-test-slave-vkjp krb5kdc[23127](info): commencing operation
krb5kdc: starting...
W20260806 04:44:03.178663 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:44:03 dist-test-slave-vkjp krb5kdc[23127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:44:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:44:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:44:03.311429 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:43231
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:43231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:44:03.429405 23141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:44:03.429688 23141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:44:03.429731 23141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:44:03.433869 23141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:44:03.433943 23141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:44:03.433964 23141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:44:03.433980 23141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:44:03.433995 23141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:44:03.439580 23141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:43231
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:43231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:44:03.440703 23141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:44:03.441771 23141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:44:03.450583 23147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:03.450636 23149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:03.450723 23141 server_base.cc:1034] running on GCE node
W20260806 04:44:03.450723 23146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:03.451586 23141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:44:03.452783 23141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:44:03.454008 23141 hybrid_clock.cc:648] HybridClock initialized: now 1785991443453991 us; error 34 us; skew 500 ppm
Aug 06 04:44:03 dist-test-slave-vkjp krb5kdc[23127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991443, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:44:03.465261 23141 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:44:03.466375 23141 webserver.cc:460] Webserver started at http://127.31.166.190:41477/ using document root <none> and password file <none>
I20260806 04:44:03.467005 23141 fs_manager.cc:359] Metadata directory not provided
I20260806 04:44:03.467128 23141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:44:03.467391 23141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:44:03.469590 23141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "56b557823004460db962ed7b4b87c79b"
format_stamp: "Formatted at 2026-08-06 04:44:03 on dist-test-slave-vkjp"
server_key: "8241e865095a5374bdbf0be4b2a5949d"
server_key_iv: "c122b00da36062905d854a42d348eda7"
server_key_version: "encryptionkey@0"
I20260806 04:44:03.470208 23141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "56b557823004460db962ed7b4b87c79b"
format_stamp: "Formatted at 2026-08-06 04:44:03 on dist-test-slave-vkjp"
server_key: "8241e865095a5374bdbf0be4b2a5949d"
server_key_iv: "c122b00da36062905d854a42d348eda7"
server_key_version: "encryptionkey@0"
I20260806 04:44:03.474081 23141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260806 04:44:03.476542 23157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:44:03.477944 23141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:44:03.478120 23141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "56b557823004460db962ed7b4b87c79b"
format_stamp: "Formatted at 2026-08-06 04:44:03 on dist-test-slave-vkjp"
server_key: "8241e865095a5374bdbf0be4b2a5949d"
server_key_iv: "c122b00da36062905d854a42d348eda7"
server_key_version: "encryptionkey@0"
I20260806 04:44:03.478251 23141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:44:03.512110 23141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:44:03.515412 23141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:44:03.515658 23141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:44:03.524132 23141 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:43231
I20260806 04:44:03.524163 23209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:43231 every 8 connection(s)
I20260806 04:44:03.525203 23141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:44:03.528129 23210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:44:03.529701 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 23141
I20260806 04:44:03.529873 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:44:03.530205 32410 external_mini_cluster.cc:1428] Setting key a86bc24f2370795e979521ce988fbeb7
I20260806 04:44:03.534719 23210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b: Bootstrap starting.
I20260806 04:44:03.537482 23210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b: Neither blocks nor log segments found. Creating new log.
I20260806 04:44:03.538406 23210 log.cc:824] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b: Log is configured to *not* fsync() on all Append() calls
I20260806 04:44:03.540313 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:03.531808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36825 (local address 127.31.166.190:43231)
0806 04:44:03.532288 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:03.532306 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:03.532343 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:44:03.533042 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:03.533059 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:03.533421 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:03.533712 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:03.534115 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:03.535342 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:03.536590 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:03.537181 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:03.537317 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:03.538336 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:03.538341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:03.538354 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:44:03.538867 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:03.538942 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:44:03.538977 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:03.539199 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:03.539347 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 04:44:03.539513 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":161,"threads_started":1}
I20260806 04:44:03.540555 23210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b: No bootstrap required, opened a new log
I20260806 04:44:03.543479 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:44:03.544440 23210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b557823004460db962ed7b4b87c79b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43231 } }
I20260806 04:44:03.544690 23210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:44:03.544780 23210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56b557823004460db962ed7b4b87c79b, State: Initialized, Role: FOLLOWER
I20260806 04:44:03.545252 23210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b557823004460db962ed7b4b87c79b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43231 } }
I20260806 04:44:03.545378 23210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:44:03.545468 23210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:44:03.545575 23210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:44:03.546604 23210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b557823004460db962ed7b4b87c79b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43231 } }
I20260806 04:44:03.546959 23210 leader_election.cc:302] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56b557823004460db962ed7b4b87c79b; no voters:
I20260806 04:44:03.547322 23210 leader_election.cc:288] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:44:03.547401 23220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:44:03.547611 23220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [term 1 LEADER]: Becoming Leader. State: Replica: 56b557823004460db962ed7b4b87c79b, State: Running, Role: LEADER
I20260806 04:44:03.547966 23220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b557823004460db962ed7b4b87c79b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43231 } }
I20260806 04:44:03.548688 23210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:44:03.549278 23220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56b557823004460db962ed7b4b87c79b. Latest consensus state: current_term: 1 leader_uuid: "56b557823004460db962ed7b4b87c79b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b557823004460db962ed7b4b87c79b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43231 } } }
I20260806 04:44:03.549423 23220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:03.550278 23221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56b557823004460db962ed7b4b87c79b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56b557823004460db962ed7b4b87c79b" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 43231 } } }
I20260806 04:44:03.550465 23221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:03.551167 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:03.545853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55189 (local address 127.31.166.190:43231)
0806 04:44:03.545991 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:03.545996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:03.546137 (+ 141us) server_negotiation.cc:407] Connection header received
0806 04:44:03.546312 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:03.546318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:03.546368 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:03.546467 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:03.546824 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:03.547710 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:03.548706 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:03.549126 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:03.549749 (+ 623us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:03.550324 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:03.550329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:03.550332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:44:03.550348 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:03.550379 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:44:03.550397 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:03.550502 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:03.550968 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 04:44:03.551008 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:44:03.551232 23227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:44:03.554571 23227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:44:03.560434 23227 catalog_manager.cc:1347] Generated new cluster ID: 3a7fb99a09014e65b5694aa7559828d5
I20260806 04:44:03.560518 23227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:44:03.584751 23227 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:44:03.586480 23227 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:44:03.593469 23227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56b557823004460db962ed7b4b87c79b: Generated new TSK 0
I20260806 04:44:03.594441 23227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:44:03.606246 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 23141
W20260806 04:44:03.612962 23125 connection.cc:535] client connection to 127.31.166.190:43231 recv error: Network error: failed to read from TLS socket (remote: 127.31.166.190:43231): Connection reset by peer (error 104)
2026-08-06T04:44:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1508 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:44:03 dist-test-slave-vkjp krb5kdc[23245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:44:03 dist-test-slave-vkjp krb5kdc[23245](info): set up 2 sockets
Aug 06 04:44:03 dist-test-slave-vkjp krb5kdc[23245](info): commencing operation
krb5kdc: starting...
W20260806 04:44:04.681720 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:44:04 dist-test-slave-vkjp krb5kdc[23245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:44:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:44:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:44:04.806001 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:41649
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:38441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:41649
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:44:04.918540 23258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:44:04.918877 23258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:44:04.918972 23258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:44:04.923413 23258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:44:04.923498 23258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:44:04.923537 23258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:44:04.923592 23258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:44:04.923628 23258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:44:04.929186 23258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:38441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:41649
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:41649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:44:04.930338 23258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:44:04.931456 23258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:44:04.939818 23268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:04.939927 23264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:04.939746 23265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:04.939791 23258 server_base.cc:1034] running on GCE node
I20260806 04:44:04.940588 23258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:44:04.941797 23258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:44:04.943002 23258 hybrid_clock.cc:648] HybridClock initialized: now 1785991444942980 us; error 35 us; skew 500 ppm
Aug 06 04:44:04 dist-test-slave-vkjp krb5kdc[23245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991444, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:44:04.945894 23258 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:44:04.946785 23258 webserver.cc:460] Webserver started at http://127.31.166.190:37335/ using document root <none> and password file <none>
I20260806 04:44:04.947422 23258 fs_manager.cc:359] Metadata directory not provided
I20260806 04:44:04.947513 23258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:44:04.947752 23258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:44:04.949898 23258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "b2d1ca719d7d4543aead74cfb2e54a16"
format_stamp: "Formatted at 2026-08-06 04:44:04 on dist-test-slave-vkjp"
server_key: "2e1615a5ede1cc29dcafb781fb858079"
server_key_iv: "6ed3f93b00acde8b7b349f9947e16f05"
server_key_version: "encryptionkey@0"
I20260806 04:44:04.950486 23258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "b2d1ca719d7d4543aead74cfb2e54a16"
format_stamp: "Formatted at 2026-08-06 04:44:04 on dist-test-slave-vkjp"
server_key: "2e1615a5ede1cc29dcafb781fb858079"
server_key_iv: "6ed3f93b00acde8b7b349f9947e16f05"
server_key_version: "encryptionkey@0"
I20260806 04:44:04.954443 23258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 04:44:04.956954 23276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:44:04.958184 23258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 04:44:04.958326 23258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "b2d1ca719d7d4543aead74cfb2e54a16"
format_stamp: "Formatted at 2026-08-06 04:44:04 on dist-test-slave-vkjp"
server_key: "2e1615a5ede1cc29dcafb781fb858079"
server_key_iv: "6ed3f93b00acde8b7b349f9947e16f05"
server_key_version: "encryptionkey@0"
I20260806 04:44:04.958444 23258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:44:04.963204 23258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:44:04.966051 23258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:44:04.966254 23258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:44:04.973285 23258 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:41649
I20260806 04:44:04.973309 23330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:41649 every 8 connection(s)
I20260806 04:44:04.974216 23258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:44:04.976925 23331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:44:04.982600 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 23258
I20260806 04:44:04.982734 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:44:04.982936 23331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16: Bootstrap starting.
I20260806 04:44:04.983064 32410 external_mini_cluster.cc:1428] Setting key 043c3f8fc7cbe603f6859dabd1afaa53
I20260806 04:44:04.985735 23331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16: Neither blocks nor log segments found. Creating new log.
I20260806 04:44:04.986642 23331 log.cc:824] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16: Log is configured to *not* fsync() on all Append() calls
I20260806 04:44:04.989121 23331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16: No bootstrap required, opened a new log
I20260806 04:44:04.989122 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:04.984510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34555 (local address 127.31.166.190:41649)
0806 04:44:04.984982 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:04.985001 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:04.985045 (+ 44us) server_negotiation.cc:407] Connection header received
0806 04:44:04.985942 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:04.985982 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:04.986346 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:04.986713 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:04.987688 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:04.987696 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:04.987713 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:44:04.987724 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:04.987780 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:44:04.987804 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:04.987914 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:04.988061 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 04:44:04.988210 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":129,"threads_started":1}
I20260806 04:44:04.991993 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:44:04.993343 23331 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d1ca719d7d4543aead74cfb2e54a16" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 41649 } }
I20260806 04:44:04.993572 23331 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:44:04.993629 23331 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d1ca719d7d4543aead74cfb2e54a16, State: Initialized, Role: FOLLOWER
I20260806 04:44:04.994136 23331 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d1ca719d7d4543aead74cfb2e54a16" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 41649 } }
I20260806 04:44:04.994311 23331 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:44:04.994383 23331 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:44:04.994549 23331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:44:04.995872 23334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:04.994111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52735 (local address 127.31.166.190:41649)
0806 04:44:04.994229 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:04.994235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:04.994436 (+ 201us) server_negotiation.cc:407] Connection header received
0806 04:44:04.994590 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:04.994592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:04.994627 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:04.994701 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:04.995475 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:04.995478 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:04.995480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:44:04.995490 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:04.995514 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:44:04.995524 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:04.995593 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:04.995712 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 04:44:04.995744 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 04:44:04.995945 23331 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d1ca719d7d4543aead74cfb2e54a16" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 41649 } }
I20260806 04:44:04.996302 23331 leader_election.cc:302] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2d1ca719d7d4543aead74cfb2e54a16; no voters:
I20260806 04:44:04.996670 23331 leader_election.cc:288] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:44:04.996737 23343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:44:04.996958 23343 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [term 1 LEADER]: Becoming Leader. State: Replica: b2d1ca719d7d4543aead74cfb2e54a16, State: Running, Role: LEADER
I20260806 04:44:04.997296 23343 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d1ca719d7d4543aead74cfb2e54a16" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 41649 } }
I20260806 04:44:04.998013 23331 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:44:04.999135 23345 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2d1ca719d7d4543aead74cfb2e54a16. Latest consensus state: current_term: 1 leader_uuid: "b2d1ca719d7d4543aead74cfb2e54a16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d1ca719d7d4543aead74cfb2e54a16" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 41649 } } }
I20260806 04:44:04.999172 23344 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2d1ca719d7d4543aead74cfb2e54a16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2d1ca719d7d4543aead74cfb2e54a16" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 41649 } } }
I20260806 04:44:04.999333 23345 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:04.999338 23344 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16 [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:04.999835 23352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:44:05.002792 23352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:44:05.008672 23352 catalog_manager.cc:1347] Generated new cluster ID: c57cd0a6f3b24c4c974575939ea82a8a
I20260806 04:44:05.008828 23352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:44:05.028252 23352 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:44:05.028631 23352 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:44:05.038201 23352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2d1ca719d7d4543aead74cfb2e54a16: Generated new TSK 0
I20260806 04:44:05.039043 23352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:44:05.049470 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 23258
2026-08-06T04:44:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:44:05 dist-test-slave-vkjp krb5kdc[23364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:44:05 dist-test-slave-vkjp krb5kdc[23364](info): set up 2 sockets
Aug 06 04:44:05 dist-test-slave-vkjp krb5kdc[23364](info): commencing operation
krb5kdc: starting...
W20260806 04:44:06.138974 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:44:06 dist-test-slave-vkjp krb5kdc[23364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:44:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:44:06.267299 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:34425
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:34425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:44:06.380012 23378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:44:06.380312 23378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:44:06.380357 23378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:44:06.384423 23378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:44:06.384485 23378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:44:06.384505 23378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:44:06.384521 23378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:44:06.384537 23378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:44:06.390558 23378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:34425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:34425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:44:06.391665 23378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:44:06.392666 23378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:44:06.401437 23384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:06.401480 23386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:06.401504 23383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:06.401656 23378 server_base.cc:1034] running on GCE node
I20260806 04:44:06.402380 23378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:44:06.403635 23378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:44:06.404830 23378 hybrid_clock.cc:648] HybridClock initialized: now 1785991446404800 us; error 50 us; skew 500 ppm
Aug 06 04:44:06 dist-test-slave-vkjp krb5kdc[23364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991446, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:44:06.407768 23378 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:44:06.408738 23378 webserver.cc:460] Webserver started at http://127.31.166.190:41769/ using document root <none> and password file <none>
I20260806 04:44:06.409309 23378 fs_manager.cc:359] Metadata directory not provided
I20260806 04:44:06.409401 23378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:44:06.409652 23378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:44:06.411810 23378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "718944c56f3e4eef9e11bae05d286311"
format_stamp: "Formatted at 2026-08-06 04:44:06 on dist-test-slave-vkjp"
server_key: "300f1e17ca7c2161e00a02924b641536"
server_key_iv: "80e11a79e86421226c67cf06a2353441"
server_key_version: "encryptionkey@0"
I20260806 04:44:06.412388 23378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "718944c56f3e4eef9e11bae05d286311"
format_stamp: "Formatted at 2026-08-06 04:44:06 on dist-test-slave-vkjp"
server_key: "300f1e17ca7c2161e00a02924b641536"
server_key_iv: "80e11a79e86421226c67cf06a2353441"
server_key_version: "encryptionkey@0"
I20260806 04:44:06.416213 23378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260806 04:44:06.418785 23393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:44:06.420492 23378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:44:06.420635 23378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "718944c56f3e4eef9e11bae05d286311"
format_stamp: "Formatted at 2026-08-06 04:44:06 on dist-test-slave-vkjp"
server_key: "300f1e17ca7c2161e00a02924b641536"
server_key_iv: "80e11a79e86421226c67cf06a2353441"
server_key_version: "encryptionkey@0"
I20260806 04:44:06.420758 23378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:44:06.439538 23378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:44:06.443336 23378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:44:06.443578 23378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:44:06.451033 23378 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:34425
I20260806 04:44:06.451148 23448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:34425 every 8 connection(s)
I20260806 04:44:06.452116 23378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:44:06.453946 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 23378
I20260806 04:44:06.454088 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:44:06.454412 32410 external_mini_cluster.cc:1428] Setting key 1a25343de0560b4bca2028b8614e3f1c
I20260806 04:44:06.455361 23449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:44:06.462123 23449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311: Bootstrap starting.
Aug 06 04:44:06 dist-test-slave-vkjp krb5kdc[23364](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:44:06.464439 23449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311: Neither blocks nor log segments found. Creating new log.
I20260806 04:44:06.465369 23449 log.cc:824] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311: Log is configured to *not* fsync() on all Append() calls
I20260806 04:44:06.467411 23449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311: No bootstrap required, opened a new log
I20260806 04:44:06.469550 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:06.455824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35499 (local address 127.31.166.190:34425)
0806 04:44:06.456245 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:06.456265 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:06.456317 (+ 52us) server_negotiation.cc:407] Connection header received
0806 04:44:06.457165 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:06.457186 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:06.457612 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:06.457965 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:06.458691 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:06.460181 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:06.461058 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:06.461736 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:06.461888 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:06.464529 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:06.464535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:06.464550 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:06.464583 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:06.466608 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:06.467123 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:06.467129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:06.467137 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:06.467258 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:06.467611 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:06.467614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:06.467616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:06.468015 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:06.468227 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:06.468536 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 04:44:06.468837 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260806 04:44:06.471513 23449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718944c56f3e4eef9e11bae05d286311" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 34425 } }
I20260806 04:44:06.471814 23449 raft_consensus.cc:374] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:44:06.471882 23449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718944c56f3e4eef9e11bae05d286311, State: Initialized, Role: FOLLOWER
I20260806 04:44:06.472339 23449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718944c56f3e4eef9e11bae05d286311" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 34425 } }
I20260806 04:44:06.472504 23449 raft_consensus.cc:388] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:44:06.472622 23449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:44:06.472682 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:44:06.472734 23449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:44:06.473675 23449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718944c56f3e4eef9e11bae05d286311" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 34425 } }
I20260806 04:44:06.474074 23449 leader_election.cc:302] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 718944c56f3e4eef9e11bae05d286311; no voters:
I20260806 04:44:06.474416 23449 leader_election.cc:288] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:44:06.474493 23459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:44:06.474677 23459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [term 1 LEADER]: Becoming Leader. State: Replica: 718944c56f3e4eef9e11bae05d286311, State: Running, Role: LEADER
I20260806 04:44:06.475066 23459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718944c56f3e4eef9e11bae05d286311" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 34425 } }
I20260806 04:44:06.475801 23449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:44:06.476329 23459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 718944c56f3e4eef9e11bae05d286311. Latest consensus state: current_term: 1 leader_uuid: "718944c56f3e4eef9e11bae05d286311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718944c56f3e4eef9e11bae05d286311" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 34425 } } }
I20260806 04:44:06.476508 23459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:06.476774 23460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "718944c56f3e4eef9e11bae05d286311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718944c56f3e4eef9e11bae05d286311" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 34425 } } }
I20260806 04:44:06.476927 23460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311 [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:06.477017 23467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:44:06.480527 23467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:44:06.484838 23453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:06.475139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59567 (local address 127.31.166.190:34425)
0806 04:44:06.475253 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:06.475257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:06.475393 (+ 136us) server_negotiation.cc:407] Connection header received
0806 04:44:06.475574 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:06.475577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:06.475626 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:06.475707 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:06.476565 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:06.477351 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:06.478595 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:06.478934 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:06.479017 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:06.480916 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:06.480919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:06.480922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:06.480948 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:06.482627 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:06.483548 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:06.483552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:06.483554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:06.483647 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:06.483935 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:06.483938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:06.483940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:06.484125 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:06.484253 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:06.484523 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 04:44:06.484646 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260806 04:44:06.486928 23467 catalog_manager.cc:1347] Generated new cluster ID: b60c679b5df945e29e093d886c307618
I20260806 04:44:06.486990 23467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:44:06.494900 23467 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:44:06.495820 23467 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:44:06.502558 23467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 718944c56f3e4eef9e11bae05d286311: Generated new TSK 0
I20260806 04:44:06.503273 23467 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:44:06.511133 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 23378
2026-08-06T04:44:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1459 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:44:06 dist-test-slave-vkjp krb5kdc[23481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:44:06 dist-test-slave-vkjp krb5kdc[23481](info): set up 2 sockets
Aug 06 04:44:06 dist-test-slave-vkjp krb5kdc[23481](info): commencing operation
krb5kdc: starting...
W20260806 04:44:07.603462 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:44:07 dist-test-slave-vkjp krb5kdc[23481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:44:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:44:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:44:07.735515 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:38415
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:46285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:38415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:44:07.849504 23494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:44:07.849836 23494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:44:07.849917 23494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:44:07.854161 23494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:44:07.854275 23494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:44:07.854316 23494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:44:07.854368 23494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:44:07.854403 23494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:44:07.860276 23494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:46285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:38415
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:38415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:44:07.861438 23494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:44:07.862483 23494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:44:07.871372 23499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:07.871372 23500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:07.871440 23502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:07.871587 23494 server_base.cc:1034] running on GCE node
I20260806 04:44:07.872243 23494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:44:07.873481 23494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:44:07.874708 23494 hybrid_clock.cc:648] HybridClock initialized: now 1785991447874693 us; error 27 us; skew 500 ppm
Aug 06 04:44:07 dist-test-slave-vkjp krb5kdc[23481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991447, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:44:07.877633 23494 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:44:07.878590 23494 webserver.cc:460] Webserver started at http://127.31.166.190:40667/ using document root <none> and password file <none>
I20260806 04:44:07.879220 23494 fs_manager.cc:359] Metadata directory not provided
I20260806 04:44:07.879314 23494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:44:07.879559 23494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:44:07.881690 23494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "e3c906f48abc4618828507295306984d"
format_stamp: "Formatted at 2026-08-06 04:44:07 on dist-test-slave-vkjp"
server_key: "8554e7178f787f42d004fb679d5774f6"
server_key_iv: "28c9c0f4e94bd003366cc3ef98abff98"
server_key_version: "encryptionkey@0"
I20260806 04:44:07.882259 23494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "e3c906f48abc4618828507295306984d"
format_stamp: "Formatted at 2026-08-06 04:44:07 on dist-test-slave-vkjp"
server_key: "8554e7178f787f42d004fb679d5774f6"
server_key_iv: "28c9c0f4e94bd003366cc3ef98abff98"
server_key_version: "encryptionkey@0"
I20260806 04:44:07.886157 23494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 04:44:07.888744 23509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:44:07.890019 23494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 04:44:07.890175 23494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "e3c906f48abc4618828507295306984d"
format_stamp: "Formatted at 2026-08-06 04:44:07 on dist-test-slave-vkjp"
server_key: "8554e7178f787f42d004fb679d5774f6"
server_key_iv: "28c9c0f4e94bd003366cc3ef98abff98"
server_key_version: "encryptionkey@0"
I20260806 04:44:07.890297 23494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:44:07.906577 23494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:44:07.909688 23494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:44:07.909929 23494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:44:07.917411 23494 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:38415
I20260806 04:44:07.917428 23565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:38415 every 8 connection(s)
I20260806 04:44:07.918393 23494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:44:07.921188 23566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:44:07.922338 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 23494
I20260806 04:44:07.922482 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:44:07.922797 32410 external_mini_cluster.cc:1428] Setting key af7ecd3da5525568fa2ed14db77d5edc
I20260806 04:44:07.927596 23566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d: Bootstrap starting.
I20260806 04:44:07.929939 23566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d: Neither blocks nor log segments found. Creating new log.
I20260806 04:44:07.930672 23566 log.cc:824] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d: Log is configured to *not* fsync() on all Append() calls
I20260806 04:44:07.931922 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:07.924239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34491 (local address 127.31.166.190:38415)
0806 04:44:07.924686 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:07.924701 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:07.924734 (+ 33us) server_negotiation.cc:407] Connection header received
0806 04:44:07.925345 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:07.925360 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:07.925696 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:07.926019 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:07.926432 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:07.927490 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:07.928560 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:07.929026 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:07.929207 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:07.930129 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:07.930134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:07.930146 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:44:07.930526 (+ 380us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:07.930580 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:44:07.930602 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:07.930709 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:07.930864 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 04:44:07.931173 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":139,"threads_started":1}
I20260806 04:44:07.932884 23566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d: No bootstrap required, opened a new log
I20260806 04:44:07.934747 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:44:07.937252 23566 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c906f48abc4618828507295306984d" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 38415 } }
I20260806 04:44:07.937575 23566 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:44:07.937682 23566 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c906f48abc4618828507295306984d, State: Initialized, Role: FOLLOWER
I20260806 04:44:07.938160 23566 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c906f48abc4618828507295306984d" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 38415 } }
I20260806 04:44:07.938295 23566 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:44:07.938365 23566 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:44:07.938488 23566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:44:07.939741 23566 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c906f48abc4618828507295306984d" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 38415 } }
I20260806 04:44:07.940140 23566 leader_election.cc:302] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3c906f48abc4618828507295306984d; no voters:
I20260806 04:44:07.940532 23566 leader_election.cc:288] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:44:07.940651 23577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:44:07.940894 23577 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [term 1 LEADER]: Becoming Leader. State: Replica: e3c906f48abc4618828507295306984d, State: Running, Role: LEADER
I20260806 04:44:07.941280 23577 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c906f48abc4618828507295306984d" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 38415 } }
I20260806 04:44:07.941913 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:07.936881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44165 (local address 127.31.166.190:38415)
0806 04:44:07.937000 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:07.937004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:07.937275 (+ 271us) server_negotiation.cc:407] Connection header received
0806 04:44:07.937386 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:07.937388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:07.937434 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:07.937515 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:07.937895 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:07.938606 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:07.939800 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:07.940119 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:07.940206 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:07.941184 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:07.941188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:07.941191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:44:07.941205 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:07.941236 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:44:07.941249 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:07.941318 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:07.941721 (+ 403us) server_negotiation.cc:299] Negotiation successful
0806 04:44:07.941788 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 04:44:07.942040 23566 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:44:07.943241 23579 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3c906f48abc4618828507295306984d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c906f48abc4618828507295306984d" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 38415 } } }
I20260806 04:44:07.943354 23579 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:07.942986 23578 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3c906f48abc4618828507295306984d. Latest consensus state: current_term: 1 leader_uuid: "e3c906f48abc4618828507295306984d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c906f48abc4618828507295306984d" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 38415 } } }
I20260806 04:44:07.943578 23578 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:07.944499 23586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:44:07.947606 23586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:44:07.952946 23586 catalog_manager.cc:1347] Generated new cluster ID: 8741e0163503451ea5b2e030cb4c197c
I20260806 04:44:07.953083 23586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:44:07.972615 23586 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:44:07.973615 23586 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:44:07.985464 23586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3c906f48abc4618828507295306984d: Generated new TSK 0
I20260806 04:44:07.986235 23586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:44:07.991888 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 23494
2026-08-06T04:44:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1481 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:44:08 dist-test-slave-vkjp krb5kdc[23600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:44:08 dist-test-slave-vkjp krb5kdc[23600](info): set up 2 sockets
Aug 06 04:44:08 dist-test-slave-vkjp krb5kdc[23600](info): commencing operation
krb5kdc: starting...
W20260806 04:44:09.114946 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:44:09 dist-test-slave-vkjp krb5kdc[23600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:44:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:44:09.258535 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:35181
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:33325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:35181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:44:09.368312 23613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:44:09.368639 23613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:44:09.368716 23613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:44:09.372742 23613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:44:09.372839 23613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:44:09.372870 23613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:44:09.372900 23613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:44:09.372938 23613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:44:09.378201 23613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:33325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:35181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:35181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:44:09.379281 23613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:44:09.380316 23613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:44:09.388767 23618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:09.388774 23621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:09.388775 23619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:09.388859 23613 server_base.cc:1034] running on GCE node
I20260806 04:44:09.389503 23613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:44:09.390578 23613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:44:09.391775 23613 hybrid_clock.cc:648] HybridClock initialized: now 1785991449391756 us; error 34 us; skew 500 ppm
Aug 06 04:44:09 dist-test-slave-vkjp krb5kdc[23600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991449, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:44:09.394526 23613 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:44:09.395426 23613 webserver.cc:460] Webserver started at http://127.31.166.190:38435/ using document root <none> and password file <none>
I20260806 04:44:09.395972 23613 fs_manager.cc:359] Metadata directory not provided
I20260806 04:44:09.396063 23613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:44:09.396297 23613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:44:09.398319 23613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "dbce364b7a024eb8a99e107c06d35186"
format_stamp: "Formatted at 2026-08-06 04:44:09 on dist-test-slave-vkjp"
server_key: "ed8825615963a367e9a05291a45cab91"
server_key_iv: "8d64f49111b3d6af7e8a1785fc144d68"
server_key_version: "encryptionkey@0"
I20260806 04:44:09.398881 23613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "dbce364b7a024eb8a99e107c06d35186"
format_stamp: "Formatted at 2026-08-06 04:44:09 on dist-test-slave-vkjp"
server_key: "ed8825615963a367e9a05291a45cab91"
server_key_iv: "8d64f49111b3d6af7e8a1785fc144d68"
server_key_version: "encryptionkey@0"
I20260806 04:44:09.402630 23613 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 04:44:09.405161 23628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:44:09.406354 23613 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 04:44:09.406503 23613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "dbce364b7a024eb8a99e107c06d35186"
format_stamp: "Formatted at 2026-08-06 04:44:09 on dist-test-slave-vkjp"
server_key: "ed8825615963a367e9a05291a45cab91"
server_key_iv: "8d64f49111b3d6af7e8a1785fc144d68"
server_key_version: "encryptionkey@0"
I20260806 04:44:09.406615 23613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:44:09.411152 23613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:44:09.414331 23613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:44:09.414527 23613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:44:09.421526 23613 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:35181
I20260806 04:44:09.421553 23683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:35181 every 8 connection(s)
I20260806 04:44:09.422410 23613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:44:09.424697 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 23613
I20260806 04:44:09.424837 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:44:09.425150 32410 external_mini_cluster.cc:1428] Setting key c7a20f4b7349894dc38a78bb8e7681bb
I20260806 04:44:09.425150 23684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:44:09.430305 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:09.426527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44653 (local address 127.31.166.190:35181)
0806 04:44:09.426956 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:09.426970 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:09.427004 (+ 34us) server_negotiation.cc:407] Connection header received
0806 04:44:09.427757 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:09.427777 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:09.428048 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:09.428373 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:09.429165 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:09.429169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:09.429180 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:44:09.429190 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:09.429241 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:44:09.429259 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:09.429445 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:09.429513 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 04:44:09.429662 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260806 04:44:09.431833 23684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186: Bootstrap starting.
I20260806 04:44:09.433159 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:44:09.434230 23684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186: Neither blocks nor log segments found. Creating new log.
I20260806 04:44:09.435144 23684 log.cc:824] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186: Log is configured to *not* fsync() on all Append() calls
I20260806 04:44:09.437141 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:09.435422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41351 (local address 127.31.166.190:35181)
0806 04:44:09.435538 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:09.435542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:09.435820 (+ 278us) server_negotiation.cc:407] Connection header received
0806 04:44:09.435970 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:09.435973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:09.436003 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:09.436098 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:09.436777 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:09.436781 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:09.436783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 04:44:09.436795 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:09.436831 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 04:44:09.436845 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:09.436935 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:09.436996 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 04:44:09.437032 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:44:09.437522 23684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186: No bootstrap required, opened a new log
I20260806 04:44:09.441082 23684 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbce364b7a024eb8a99e107c06d35186" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35181 } }
I20260806 04:44:09.441287 23684 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 04:44:09.441331 23684 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbce364b7a024eb8a99e107c06d35186, State: Initialized, Role: FOLLOWER
I20260806 04:44:09.441674 23684 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbce364b7a024eb8a99e107c06d35186" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35181 } }
I20260806 04:44:09.441777 23684 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 04:44:09.441817 23684 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 04:44:09.441885 23684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:44:09.442795 23684 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbce364b7a024eb8a99e107c06d35186" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35181 } }
I20260806 04:44:09.443193 23684 leader_election.cc:302] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbce364b7a024eb8a99e107c06d35186; no voters:
I20260806 04:44:09.443531 23684 leader_election.cc:288] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 04:44:09.443656 23694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:44:09.443897 23694 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [term 1 LEADER]: Becoming Leader. State: Replica: dbce364b7a024eb8a99e107c06d35186, State: Running, Role: LEADER
I20260806 04:44:09.444180 23694 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbce364b7a024eb8a99e107c06d35186" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35181 } }
I20260806 04:44:09.444737 23684 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:44:09.445832 23695 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbce364b7a024eb8a99e107c06d35186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbce364b7a024eb8a99e107c06d35186" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35181 } } }
I20260806 04:44:09.445971 23695 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:09.446249 23696 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbce364b7a024eb8a99e107c06d35186. Latest consensus state: current_term: 1 leader_uuid: "dbce364b7a024eb8a99e107c06d35186" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbce364b7a024eb8a99e107c06d35186" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 35181 } } }
I20260806 04:44:09.446341 23696 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186 [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:09.446475 23704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:44:09.449553 23704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:44:09.454809 23704 catalog_manager.cc:1347] Generated new cluster ID: 96db5c146de443d48ccb246c3fa82b63
I20260806 04:44:09.454893 23704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:44:09.477960 23704 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:44:09.478338 23704 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:44:09.485769 23704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbce364b7a024eb8a99e107c06d35186: Generated new TSK 0
I20260806 04:44:09.486490 23704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 04:44:09.490649 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 23613
2026-08-06T04:44:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1497 ms)
[----------] 18 tests from AuthTokenIssuingTest (26695 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:44:09 dist-test-slave-vkjp krb5kdc[23718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:44:09 dist-test-slave-vkjp krb5kdc[23718](info): set up 2 sockets
Aug 06 04:44:09 dist-test-slave-vkjp krb5kdc[23718](info): commencing operation
krb5kdc: starting...
W20260806 04:44:10.558818 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:44:10 dist-test-slave-vkjp krb5kdc[23718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:44:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:44:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:44:10.687182 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:39357
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:39357,127.31.166.189:36889,127.31.166.188:38507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:44:10.798564 23731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:44:10.798924 23731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:44:10.799000 23731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:44:10.799096 23731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:44:10.803459 23731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:44:10.803568 23731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:44:10.803608 23731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:44:10.803647 23731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:44:10.803679 23731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:44:10.809033 23731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:39357,127.31.166.189:36889,127.31.166.188:38507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:39357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:44:10.810099 23731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:44:10.811144 23731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:44:10.819510 23737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:10.819540 23740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:10.819522 23738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:10.819921 23731 server_base.cc:1034] running on GCE node
I20260806 04:44:10.820456 23731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:44:10.821585 23731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:44:10.822769 23731 hybrid_clock.cc:648] HybridClock initialized: now 1785991450822735 us; error 56 us; skew 500 ppm
Aug 06 04:44:10 dist-test-slave-vkjp krb5kdc[23718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991450, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:44:10.825575 23731 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:44:10.826493 23731 webserver.cc:460] Webserver started at http://127.31.166.190:46713/ using document root <none> and password file <none>
I20260806 04:44:10.827050 23731 fs_manager.cc:359] Metadata directory not provided
I20260806 04:44:10.827190 23731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:44:10.827437 23731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:44:10.829460 23731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "f8b5ec6069d449ac8d272eb702b8b5e7"
format_stamp: "Formatted at 2026-08-06 04:44:10 on dist-test-slave-vkjp"
server_key: "00bcd4b32d3d3f7eda80b9505c02d6aa"
server_key_iv: "54ee2b95ffcfe74c0306a3ba7e2c6edd"
server_key_version: "encryptionkey@0"
I20260806 04:44:10.830034 23731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "f8b5ec6069d449ac8d272eb702b8b5e7"
format_stamp: "Formatted at 2026-08-06 04:44:10 on dist-test-slave-vkjp"
server_key: "00bcd4b32d3d3f7eda80b9505c02d6aa"
server_key_iv: "54ee2b95ffcfe74c0306a3ba7e2c6edd"
server_key_version: "encryptionkey@0"
I20260806 04:44:10.833726 23731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260806 04:44:10.836117 23748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:44:10.837288 23731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:44:10.837446 23731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "f8b5ec6069d449ac8d272eb702b8b5e7"
format_stamp: "Formatted at 2026-08-06 04:44:10 on dist-test-slave-vkjp"
server_key: "00bcd4b32d3d3f7eda80b9505c02d6aa"
server_key_iv: "54ee2b95ffcfe74c0306a3ba7e2c6edd"
server_key_version: "encryptionkey@0"
I20260806 04:44:10.837558 23731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:44:10.847853 23731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:44:10.850803 23731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:44:10.851011 23731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:44:10.858034 23731 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:39357
I20260806 04:44:10.858399 23801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:39357 every 8 connection(s)
I20260806 04:44:10.858968 23731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:44:10.861624 23802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:44:10.863737 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 23731
I20260806 04:44:10.863881 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:44:10.864169 32410 external_mini_cluster.cc:1428] Setting key 2a96fe9907171554f0aa937a7628fc80
I20260806 04:44:10.866218 23802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.166.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.189@KRBTEST.COM" created.
Aug 06 04:44:10 dist-test-slave-vkjp krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991450, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:44:10.881585 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:10.869568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.31.166.190:39357)
0806 04:44:10.869896 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:10.869905 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:10.869948 (+ 43us) server_negotiation.cc:407] Connection header received
0806 04:44:10.870374 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:10.870378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:10.870544 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:10.870719 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:10.871553 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:10.872440 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:10.873478 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:10.873841 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:10.874050 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:10.876752 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:10.876762 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:10.876786 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:10.876820 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:10.878450 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:10.879062 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:10.879066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:10.879067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:10.879225 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:10.879615 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:10.879619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:10.879623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:10.880006 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:10.880112 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:10.880609 (+ 497us) server_negotiation.cc:299] Negotiation successful
0806 04:44:10.880821 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":222,"thread_start_us":92,"threads_started":1}
I20260806 04:44:10.881588 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:10.868579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39357 (local address 127.0.0.1:59642)
0806 04:44:10.869112 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:10.869154 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:10.870086 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:10.870719 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:10.870723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:10.871179 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:10.871411 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:10.871423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:10.872577 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:10.872580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:10.873295 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:10.873303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:10.874035 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:10.874039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:10.874247 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:10.874873 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:10.874888 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:10.876599 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:10.878568 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:10.878581 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:10.878594 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:10.878922 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:10.879327 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:10.879334 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:10.879338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:10.879500 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:10.880137 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:10.880146 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:10.880334 (+ 188us) client_negotiation.cc:769] Sending connection context
0806 04:44:10.880545 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 04:44:10.880939 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":138,"thread_start_us":143,"threads_started":1}
I20260806 04:44:10.884294 23802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:44:10.885061 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:10.884767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:36889 (local address 127.0.0.1:55370)
0806 04:44:10.884886 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:10.884983 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:36889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 04:44:10.885263 23752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.166.189:36889: connect: Connection refused (error 111)
I20260806 04:44:10.886108 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:10.885880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:36889 (local address 127.0.0.1:55374)
0806 04:44:10.886005 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:10.886059 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:36889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 04:44:10.886348 23802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.189:36889: Network error: Client connection negotiation failed: client connection to 127.31.166.189:36889: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.189.keytab.
Entry for principal kudu/127.31.166.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.189.keytab.
I20260806 04:44:10.894466 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.189:36889
--webserver_interface=127.31.166.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.189.keytab
--principal=kudu/127.31.166.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:39357,127.31.166.189:36889,127.31.166.188:38507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:44:10.935708 23802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } attempt: 1
I20260806 04:44:10.936677 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:10.936308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:36889 (local address 127.0.0.1:55388)
0806 04:44:10.936501 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:10.936577 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:36889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 04:44:10.937508 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:10.937266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:36889 (local address 127.0.0.1:55394)
0806 04:44:10.937383 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:10.937439 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:36889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:44:10.937698 23802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.189:36889: Network error: Client connection negotiation failed: client connection to 127.31.166.189:36889: connect: Connection refused (error 111)
I20260806 04:44:11.005928 23802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } attempt: 2
W20260806 04:44:11.005805 23809 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:44:11.006170 23809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:44:11.006233 23809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:44:11.006342 23809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 04:44:11.006892 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.006519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:36889 (local address 127.0.0.1:55396)
0806 04:44:11.006705 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.006784 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:36889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 04:44:11.007768 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.007511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:36889 (local address 127.0.0.1:55410)
0806 04:44:11.007638 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.007696 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:36889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 04:44:11.007944 23802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.189:36889: Network error: Client connection negotiation failed: client connection to 127.31.166.189:36889: connect: Connection refused (error 111)
W20260806 04:44:11.011572 23809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:44:11.011629 23809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:44:11.011652 23809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:44:11.011672 23809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:44:11.011731 23809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:44:11.017377 23809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:39357,127.31.166.189:36889,127.31.166.188:38507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.189.keytab
--principal=kudu/127.31.166.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.189:36889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.166.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:44:11.018446 23809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:44:11.019528 23809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:44:11.027796 23817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:11.027808 23816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:11.027808 23819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:11.028038 23809 server_base.cc:1034] running on GCE node
I20260806 04:44:11.028628 23809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:44:11.029719 23809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:44:11.030922 23809 hybrid_clock.cc:648] HybridClock initialized: now 1785991451030892 us; error 48 us; skew 500 ppm
Aug 06 04:44:11 dist-test-slave-vkjp krb5kdc[23718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991451, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:44:11.033840 23809 init.cc:377] Logged in from keytab as kudu/127.31.166.189@KRBTEST.COM (short username kudu)
I20260806 04:44:11.034888 23809 webserver.cc:460] Webserver started at http://127.31.166.189:33171/ using document root <none> and password file <none>
I20260806 04:44:11.035576 23809 fs_manager.cc:359] Metadata directory not provided
I20260806 04:44:11.035686 23809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:44:11.035952 23809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:44:11.038254 23809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/data/instance:
uuid: "3a64e02b164a435c8a81dd2eeb6eb59e"
format_stamp: "Formatted at 2026-08-06 04:44:11 on dist-test-slave-vkjp"
server_key: "e523f087092a8a46eeea6bf3596c8774"
server_key_iv: "09c5a327c36c0f8b5ef198247b2257d6"
server_key_version: "encryptionkey@0"
I20260806 04:44:11.038976 23809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/wal/instance:
uuid: "3a64e02b164a435c8a81dd2eeb6eb59e"
format_stamp: "Formatted at 2026-08-06 04:44:11 on dist-test-slave-vkjp"
server_key: "e523f087092a8a46eeea6bf3596c8774"
server_key_iv: "09c5a327c36c0f8b5ef198247b2257d6"
server_key_version: "encryptionkey@0"
I20260806 04:44:11.043459 23809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 04:44:11.046180 23826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:44:11.047530 23809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:44:11.047668 23809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/wal
uuid: "3a64e02b164a435c8a81dd2eeb6eb59e"
format_stamp: "Formatted at 2026-08-06 04:44:11 on dist-test-slave-vkjp"
server_key: "e523f087092a8a46eeea6bf3596c8774"
server_key_iv: "09c5a327c36c0f8b5ef198247b2257d6"
server_key_version: "encryptionkey@0"
I20260806 04:44:11.047780 23809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:44:11.063783 23809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:44:11.066689 23809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:44:11.066921 23809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:44:11.074920 23809 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.189:36889
I20260806 04:44:11.075255 23882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.189:36889 every 8 connection(s)
I20260806 04:44:11.076186 23809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/data/info.pb
I20260806 04:44:11.078996 23883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:44:11.081521 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 23809
I20260806 04:44:11.081663 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-1/wal/instance
I20260806 04:44:11.081938 32410 external_mini_cluster.cc:1428] Setting key cf09daad2300a06cc4c041d97346ad5e
I20260806 04:44:11.082779 23883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.166.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.188@KRBTEST.COM" created.
Aug 06 04:44:11 dist-test-slave-vkjp krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991451, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.189@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:44:11.099144 23802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } attempt: 3
I20260806 04:44:11.100860 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.084656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.31.166.190:39357)
0806 04:44:11.084818 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.084823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.085638 (+ 815us) server_negotiation.cc:407] Connection header received
0806 04:44:11.086484 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.086490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.086557 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.086655 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.087941 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.088924 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.090051 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.090442 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.090568 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.095725 (+ 5157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:11.095732 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:11.095738 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:11.095786 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:11.098282 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.098983 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.098987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.098989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.099133 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.099552 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.099558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.099563 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.099805 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:11.099904 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:11.100548 (+ 644us) server_negotiation.cc:299] Negotiation successful
0806 04:44:11.100677 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 04:44:11.101753 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.085061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39357 (local address 127.0.0.1:59646)
0806 04:44:11.085484 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.085525 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:11.086287 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:11.086850 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:11.086857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:11.087287 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:11.087769 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.087782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.089075 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.089079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.089920 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.089929 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.090604 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.090607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.090746 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.091426 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:11.091452 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:11.095532 (+ 4080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:11.098411 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.098421 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.098440 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.098858 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.099225 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.099232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.099236 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.099436 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.099959 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:11.099965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:11.100289 (+ 324us) client_negotiation.cc:769] Sending connection context
0806 04:44:11.100495 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 04:44:11.100817 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
I20260806 04:44:11.103303 23883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:44:11 dist-test-slave-vkjp krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991450, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for kudu/127.31.166.189@KRBTEST.COM
Aug 06 04:44:11 dist-test-slave-vkjp krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991451, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.189@KRBTEST.COM for kudu/127.31.166.189@KRBTEST.COM
I20260806 04:44:11.111877 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.099665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:36889 (local address 127.0.0.1:55418)
0806 04:44:11.099861 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.099878 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:11.099969 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:11.100835 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:11.100841 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:11.101059 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:11.101200 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.101209 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.102767 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.102770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.103536 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.103546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.104755 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.104758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.104843 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.105422 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:11.105434 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:11.107041 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:11.109281 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.109285 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.109288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.109579 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.110032 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.110035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.110037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.110142 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.110967 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:11.110971 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:11.111191 (+ 220us) client_negotiation.cc:769] Sending connection context
0806 04:44:11.111376 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 04:44:11.111570 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 04:44:11.112195 23888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.099896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55418 (local address 127.31.166.189:36889)
0806 04:44:11.100275 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.100289 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.100314 (+ 25us) server_negotiation.cc:407] Connection header received
0806 04:44:11.100364 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.100373 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.100647 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.101378 (+ 731us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.101512 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.102660 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.104171 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.104572 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.104688 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.107194 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:11.107207 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:11.107224 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:11.107258 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:11.109140 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.109752 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.109761 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.109764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.109894 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.110359 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.110362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.110363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.110807 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:11.111030 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:11.111570 (+ 540us) server_negotiation.cc:299] Negotiation successful
0806 04:44:11.111929 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"spinlock_wait_cycles":3712,"thread_start_us":151,"threads_started":1}
I20260806 04:44:11.113220 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.103828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:36889 (local address 127.0.0.1:55424)
0806 04:44:11.103935 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.103949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:11.104019 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:11.104535 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:11.104540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:11.104717 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:11.104824 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.104830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.105631 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.105638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.106141 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.106148 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.106596 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.106600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.106684 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.107225 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:11.107250 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:11.108825 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:11.111565 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.111571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.111573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.111855 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.112136 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.112139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.112141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.112319 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.112711 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:11.112714 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:11.112807 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 04:44:11.112947 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 04:44:11.113094 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"spinlock_wait_cycles":1280}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.188.keytab.
Entry for principal kudu/127.31.166.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.188.keytab.
I20260806 04:44:11.113581 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.103981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.31.166.189:36889)
0806 04:44:11.104292 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.104302 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.104318 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:44:11.104357 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.104362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.104411 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.104484 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.104943 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.105539 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.106255 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.106491 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.106597 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.108980 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:11.108986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:11.108989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:11.109018 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:11.111460 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.111953 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.111958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.111960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.112041 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.112428 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.112430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.112432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.112622 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:11.112718 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:11.113019 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 04:44:11.113165 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":233,"thread_start_us":125,"threads_started":1}
I20260806 04:44:11.114593 23802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:44:11.114754 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.188:38507
--webserver_interface=127.31.166.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.188.keytab
--principal=kudu/127.31.166.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:39357,127.31.166.189:36889,127.31.166.188:38507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:44:11.115381 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.115098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50164)
0806 04:44:11.115250 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.115311 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 04:44:11.116252 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.115902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50172)
0806 04:44:11.116069 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.116163 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260806 04:44:11.116436 23802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:38507: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
I20260806 04:44:11.116926 23883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:44:11.117728 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.117381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50180)
0806 04:44:11.117535 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.117639 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 04:44:11.117975 23830 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
I20260806 04:44:11.118866 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.118648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50196)
0806 04:44:11.118769 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.118815 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 04:44:11.119159 23883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:38507: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
I20260806 04:44:11.147606 23802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } attempt: 1
I20260806 04:44:11.148581 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.148252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50198)
0806 04:44:11.148428 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.148496 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 04:44:11.149385 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.149110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50212)
0806 04:44:11.149240 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.149326 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260806 04:44:11.149586 23802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:38507: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
I20260806 04:44:11.168484 23883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } attempt: 1
I20260806 04:44:11.169349 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.169052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50220)
0806 04:44:11.169198 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.169267 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 04:44:11.170100 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.169884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50236)
0806 04:44:11.169997 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.170049 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 04:44:11.170310 23883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:38507: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
I20260806 04:44:11.224792 23802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } attempt: 2
I20260806 04:44:11.225746 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.225405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50248)
0806 04:44:11.225569 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.225660 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260806 04:44:11.226052 23890 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:44:11.226405 23890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:44:11.226457 23890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:44:11.226543 23890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 04:44:11.226730 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.226399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50262)
0806 04:44:11.226547 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.226650 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 04:44:11.226986 23802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:38507: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
W20260806 04:44:11.230679 23890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:44:11.230731 23890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:44:11.230751 23890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:44:11.230767 23890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:44:11.230813 23890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:44:11.236047 23890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:39357,127.31.166.189:36889,127.31.166.188:38507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.188.keytab
--principal=kudu/127.31.166.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.188:38507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.166.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:44:11.237052 23890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:44:11.238041 23890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 04:44:11.238498 23883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } attempt: 2
I20260806 04:44:11.239410 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.239063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50266)
0806 04:44:11.239255 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.239318 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260806 04:44:11.240193 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.239960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50274)
0806 04:44:11.240074 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.240128 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 04:44:11.240360 23883 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:38507: Network error: Client connection negotiation failed: client connection to 127.31.166.188:38507: connect: Connection refused (error 111)
W20260806 04:44:11.246951 23900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:11.247002 23898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:11.247148 23897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:11.247383 23890 server_base.cc:1034] running on GCE node
I20260806 04:44:11.247964 23890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:44:11.249132 23890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:44:11.250325 23890 hybrid_clock.cc:648] HybridClock initialized: now 1785991451250292 us; error 41 us; skew 500 ppm
Aug 06 04:44:11 dist-test-slave-vkjp krb5kdc[23718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991451, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:44:11.253317 23890 init.cc:377] Logged in from keytab as kudu/127.31.166.188@KRBTEST.COM (short username kudu)
I20260806 04:44:11.254141 23890 webserver.cc:460] Webserver started at http://127.31.166.188:33283/ using document root <none> and password file <none>
I20260806 04:44:11.254637 23890 fs_manager.cc:359] Metadata directory not provided
I20260806 04:44:11.254694 23890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:44:11.254863 23890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:44:11.256875 23890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/data/instance:
uuid: "e5368d3c60f04db498e809dba30d328d"
format_stamp: "Formatted at 2026-08-06 04:44:11 on dist-test-slave-vkjp"
server_key: "e8374fc05efdf2f0aa69a4f9b657f514"
server_key_iv: "ee8c50f18d143d555ade4d6f8080cb98"
server_key_version: "encryptionkey@0"
I20260806 04:44:11.257377 23890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/wal/instance:
uuid: "e5368d3c60f04db498e809dba30d328d"
format_stamp: "Formatted at 2026-08-06 04:44:11 on dist-test-slave-vkjp"
server_key: "e8374fc05efdf2f0aa69a4f9b657f514"
server_key_iv: "ee8c50f18d143d555ade4d6f8080cb98"
server_key_version: "encryptionkey@0"
I20260806 04:44:11.261112 23890 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 04:44:11.263722 23907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:44:11.265035 23890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260806 04:44:11.265147 23890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/wal
uuid: "e5368d3c60f04db498e809dba30d328d"
format_stamp: "Formatted at 2026-08-06 04:44:11 on dist-test-slave-vkjp"
server_key: "e8374fc05efdf2f0aa69a4f9b657f514"
server_key_iv: "ee8c50f18d143d555ade4d6f8080cb98"
server_key_version: "encryptionkey@0"
I20260806 04:44:11.265230 23890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:44:11.282938 23890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:44:11.286417 23890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:44:11.286577 23890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:44:11.293691 23890 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.188:38507
I20260806 04:44:11.293984 23960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.188:38507 every 8 connection(s)
I20260806 04:44:11.294718 23890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/data/info.pb
I20260806 04:44:11.297470 23961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:44:11.301146 23961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:44:11.302682 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 23890
I20260806 04:44:11.302824 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785991160137184-32410-0/minicluster-data/master-2/wal/instance
I20260806 04:44:11.303177 32410 external_mini_cluster.cc:1428] Setting key c21d65ea74d7d8da80438ed39c7ddf3e
Aug 06 04:44:11 dist-test-slave-vkjp krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991451, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.188@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
Aug 06 04:44:11 dist-test-slave-vkjp krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.188@KRBTEST.COM
I20260806 04:44:11.315184 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.303238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59662 (local address 127.31.166.190:39357)
0806 04:44:11.303385 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.303388 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.304287 (+ 899us) server_negotiation.cc:407] Connection header received
0806 04:44:11.305383 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.305386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.305434 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.305504 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.306409 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.306914 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.308079 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.308373 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.308455 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.311228 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:11.311233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:11.311236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:11.311283 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:11.312872 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.313451 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.313454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.313455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.313529 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.313903 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.313907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.313908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.314165 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:11.314266 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:11.314846 (+ 580us) server_negotiation.cc:299] Negotiation successful
0806 04:44:11.314971 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 04:44:11.316179 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.303656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:39357 (local address 127.0.0.1:59662)
0806 04:44:11.304128 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.304185 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:11.305163 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:11.305573 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:11.305578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:11.305996 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:11.306235 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.306247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.307023 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.307026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.307947 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.307955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.308507 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.308510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.308680 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.309498 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:11.309513 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:11.311033 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:11.312990 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.312996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.313005 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.313338 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.313614 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.313620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.313625 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.313792 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.314268 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:11.314274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:11.314604 (+ 330us) client_negotiation.cc:769] Sending connection context
0806 04:44:11.314784 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 04:44:11.315137 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":76,"thread_start_us":117,"threads_started":1}
I20260806 04:44:11.316179 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.304529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50284 (local address 127.31.166.188:38507)
0806 04:44:11.304790 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.304804 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.304843 (+ 39us) server_negotiation.cc:407] Connection header received
0806 04:44:11.305163 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.305185 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.305407 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.305604 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.306326 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.307383 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.308246 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.308665 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.308906 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.311675 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:11.311683 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:11.311699 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:11.311726 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:11.313582 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.314010 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.314014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.314015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.314084 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.314374 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.314377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.314378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.314663 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:11.314877 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:11.315168 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 04:44:11.315314 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260806 04:44:11.318037 23961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:44:11 dist-test-slave-vkjp krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991451, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.188@KRBTEST.COM for kudu/127.31.166.189@KRBTEST.COM
Aug 06 04:44:11 dist-test-slave-vkjp krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:44:11.326154 23802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } attempt: 3
I20260806 04:44:11.327682 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.318587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:36889 (local address 127.0.0.1:55434)
0806 04:44:11.318821 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.318833 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:11.318909 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:11.319260 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:11.319262 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:11.319389 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:11.319480 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.319485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.320328 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.320334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.320890 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.320898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.321482 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.321486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.321601 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.322146 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:11.322198 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:11.323703 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:11.325785 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.325787 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.325789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.325982 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.326380 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.326384 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.326387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.326486 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.327093 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:11.327122 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:11.327221 (+ 99us) client_negotiation.cc:769] Sending connection context
0806 04:44:11.327373 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 04:44:11.327511 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260806 04:44:11.327773 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.318877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.31.166.189:36889)
0806 04:44:11.318988 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.318992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.319007 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:44:11.319055 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.319058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.319144 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.319235 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.319603 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.320216 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.321029 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.321360 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.321444 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.323845 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:11.323854 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:11.323856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:11.323906 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:11.325672 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.326129 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.326135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.326137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.326227 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.326740 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.326747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.326749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.326945 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:11.327003 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:11.327459 (+ 456us) server_negotiation.cc:299] Negotiation successful
0806 04:44:11.327602 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 04:44:11.328249 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.318588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.31.166.190:39357)
0806 04:44:11.318708 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.318712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.318731 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:44:11.318907 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.318910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.318958 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.319034 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.319809 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.320308 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.321230 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.321509 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.321619 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.324376 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:11.324379 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:11.324385 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:11.324413 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:11.325958 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.326405 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.326408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.326410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.326486 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.326779 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.326783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.326784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.327015 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:11.327281 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:11.327969 (+ 688us) server_negotiation.cc:299] Negotiation successful
0806 04:44:11.328114 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 04:44:11.328698 23961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:44:11.331517 23883 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } attempt: 3
Aug 06 04:44:11 dist-test-slave-vkjp krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991450, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for kudu/127.31.166.188@KRBTEST.COM
Aug 06 04:44:11 dist-test-slave-vkjp krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991451, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.188@KRBTEST.COM for kudu/127.31.166.188@KRBTEST.COM
Aug 06 04:44:11 dist-test-slave-vkjp krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.189@KRBTEST.COM
I20260806 04:44:11.336344 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.326622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50298)
0806 04:44:11.326728 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.326744 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:11.326826 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:11.327563 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:11.327568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:11.327728 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:11.327843 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.327851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.329154 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.329159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.329841 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.329851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.330514 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.330517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.330603 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.331077 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:11.331120 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:11.332472 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:11.334462 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.334467 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.334469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.334729 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.335103 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.335109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.335111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.335217 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.335728 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:11.335733 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:11.335828 (+ 95us) client_negotiation.cc:769] Sending connection context
0806 04:44:11.335963 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 04:44:11.336116 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260806 04:44:11.336918 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.326670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.31.166.188:38507)
0806 04:44:11.327329 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.327333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.327345 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:44:11.327381 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.327386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.327430 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.328260 (+ 830us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.328352 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.329034 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.329994 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.330404 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.330511 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.332608 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:11.332612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:11.332615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:11.332640 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:11.334368 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.334842 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.334847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.334848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.334941 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.335338 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.335341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.335343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.335609 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:11.335722 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:11.336575 (+ 853us) server_negotiation.cc:299] Negotiation successful
0806 04:44:11.336705 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575}
Aug 06 04:44:11 dist-test-slave-vkjp krb5kdc[23718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991451, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.189@KRBTEST.COM for kudu/127.31.166.188@KRBTEST.COM
I20260806 04:44:11.339633 23963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.329132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50300)
0806 04:44:11.329374 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.329390 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:11.329461 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:11.329933 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:11.329938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:11.330107 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:11.330183 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.330189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.331754 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.331760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.332508 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.332517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.333069 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.333072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.333149 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.333796 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:11.333811 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:11.335760 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:11.338135 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.338139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.338141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.338333 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.338570 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.338574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.338575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.338656 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.339166 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:11.339172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:11.339257 (+ 85us) client_negotiation.cc:769] Sending connection context
0806 04:44:11.339341 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 04:44:11.339482 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260806 04:44:11.340162 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.329397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.31.166.188:38507)
0806 04:44:11.329705 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.329716 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.329731 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:44:11.329765 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.329768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.329810 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.329895 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.330327 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.331636 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.332628 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.332968 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.334474 (+ 1506us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.335914 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:11.335920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:11.335926 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:11.335955 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:11.338032 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.338420 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.338422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.338423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.338477 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.338747 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.338753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.338755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.339031 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:11.339880 (+ 849us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:11.339949 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 04:44:11.340047 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
I20260806 04:44:11.341043 23802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7: Bootstrap starting.
I20260806 04:44:11.342365 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.329502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.31.166.189:36889)
0806 04:44:11.329610 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.329615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.329768 (+ 153us) server_negotiation.cc:407] Connection header received
0806 04:44:11.329946 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.329949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.329991 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.330051 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.330931 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.332169 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.333214 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.333574 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.333667 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.337235 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:11.337239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:11.337242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:11.337267 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:11.339666 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.340921 (+ 1255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.340924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.340926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.341031 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.341365 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.341368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.341370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.341568 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:11.341730 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:11.342104 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 04:44:11.342218 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 04:44:11.343159 23961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d: Bootstrap starting.
I20260806 04:44:11.344333 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.331880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:38507 (local address 127.0.0.1:50306)
0806 04:44:11.332030 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:11.332043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:11.332109 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:11.332982 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:11.332986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:11.333184 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:11.333301 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.333309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.336044 (+ 2735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.336048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.336906 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:11.336921 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.337551 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.337555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:11.337646 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.338285 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:11.338299 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:11.340025 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:11.342462 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.342465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.342467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.342706 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.343039 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:11.343042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:11.343044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:11.343166 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:11.343842 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:11.343857 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:11.343945 (+ 88us) client_negotiation.cc:769] Sending connection context
0806 04:44:11.344027 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 04:44:11.344151 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 04:44:11.344327 23802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7: Neither blocks nor log segments found. Creating new log.
I20260806 04:44:11.345135 23967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.331936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50306 (local address 127.31.166.188:38507)
0806 04:44:11.332718 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.332728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.332742 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:44:11.332775 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.332780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.332838 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.333420 (+ 582us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.333560 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.335921 (+ 2361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.337064 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.337409 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.340154 (+ 2745us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.340255 (+ 101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:11.340257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:11.340260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:11.340275 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:11.342338 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.342827 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.342830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.342831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.342927 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.343529 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.343532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.343534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.343725 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:11.344791 (+ 1066us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:11.344879 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 04:44:11.344986 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1515,"server-negotiator.queue_time_us":719,"thread_start_us":101,"threads_started":1}
I20260806 04:44:11.345209 23802 log.cc:824] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7: Log is configured to *not* fsync() on all Append() calls
I20260806 04:44:11.345646 23961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d: Neither blocks nor log segments found. Creating new log.
I20260806 04:44:11.346350 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:44:11.346486 23961 log.cc:824] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d: Log is configured to *not* fsync() on all Append() calls
I20260806 04:44:11.347455 23802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7: No bootstrap required, opened a new log
I20260806 04:44:11.348265 23883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e: Bootstrap starting.
I20260806 04:44:11.348574 23961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d: No bootstrap required, opened a new log
I20260806 04:44:11.350576 23883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e: Neither blocks nor log segments found. Creating new log.
I20260806 04:44:11.350987 23775 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59672
I20260806 04:44:11.351415 23883 log.cc:824] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e: Log is configured to *not* fsync() on all Append() calls
I20260806 04:44:11.351670 23802 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } }
I20260806 04:44:11.351940 23802 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b5ec6069d449ac8d272eb702b8b5e7, State: Initialized, Role: FOLLOWER
I20260806 04:44:11.352264 23802 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } }
I20260806 04:44:11.352260 23961 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } }
I20260806 04:44:11.352546 23961 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5368d3c60f04db498e809dba30d328d, State: Initialized, Role: FOLLOWER
I20260806 04:44:11.352807 23973 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } } }
I20260806 04:44:11.352866 23961 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } }
I20260806 04:44:11.353062 23973 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:44:11.353758 23802 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:44:11.353822 23775 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:44:11.353971 23883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e: No bootstrap required, opened a new log
I20260806 04:44:11.353950 23775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:44:11.354105 23961 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:44:11.355386 23775 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } }
I20260806 04:44:11.356343 23775 leader_election.cc:288] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [CANDIDATE]: Term 1 election: Requested vote from peers 3a64e02b164a435c8a81dd2eeb6eb59e (127.31.166.189:36889), e5368d3c60f04db498e809dba30d328d (127.31.166.188:38507)
I20260806 04:44:11.357188 23934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e5368d3c60f04db498e809dba30d328d"
I20260806 04:44:11.357415 23934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:44:11.358795 23934 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8b5ec6069d449ac8d272eb702b8b5e7 in term 1.
I20260806 04:44:11.359244 23752 leader_election.cc:302] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e5368d3c60f04db498e809dba30d328d, f8b5ec6069d449ac8d272eb702b8b5e7; no voters:
I20260806 04:44:11.359529 23973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:44:11.359737 23973 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [term 1 LEADER]: Becoming Leader. State: Replica: f8b5ec6069d449ac8d272eb702b8b5e7, State: Running, Role: LEADER
I20260806 04:44:11.360065 23973 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } }
I20260806 04:44:11.361898 23973 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b5ec6069d449ac8d272eb702b8b5e7. Latest consensus state: current_term: 1 leader_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } } }
I20260806 04:44:11.362032 23973 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [sys.catalog]: This master's current role is: LEADER
W20260806 04:44:11.362216 23993 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:44:11.362340 23993 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:44:11.362612 23974 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } } }
I20260806 04:44:11.362744 23974 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:44:11.358917 23731 master_runner.cc:186] Error getting master registration for 127.31.166.189:36889: OK, instance_id { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" instance_seqno: 1785991451072193 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:44:11.362926 23731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:44:11.365533 23883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } }
W20260806 04:44:11.365676 23996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:44:11.365736 23996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:44:11.365761 23883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a64e02b164a435c8a81dd2eeb6eb59e, State: Initialized, Role: FOLLOWER
I20260806 04:44:11.366093 23883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } }
I20260806 04:44:11.367455 23883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:44:11.366395 23890 master_runner.cc:186] Error getting master registration for 127.31.166.189:36889: OK, instance_id { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" instance_seqno: 1785991451072193 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:44:11.369843 23890 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:44:11.370534 24006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } } }
I20260806 04:44:11.370656 24006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:44:11.370132 23855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e"
I20260806 04:44:11.370859 23855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:44:11.372370 23855 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8b5ec6069d449ac8d272eb702b8b5e7 in term 1.
I20260806 04:44:11.374579 24004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:44:11.375710 24004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260806 04:44:11.378249 24017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:44:11.378340 24017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:44:11.387784 23934 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d [term 1 FOLLOWER]: Refusing update from remote peer f8b5ec6069d449ac8d272eb702b8b5e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:44:11.388538 24002 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:44:11.392442 23974 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b5ec6069d449ac8d272eb702b8b5e7. Latest consensus state: current_term: 1 leader_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } } }
I20260806 04:44:11.392575 23974 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:44:11.393494 23855 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e [term 1 FOLLOWER]: Refusing update from remote peer f8b5ec6069d449ac8d272eb702b8b5e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:44:11.394096 24002 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:44:11.398808 24006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b5ec6069d449ac8d272eb702b8b5e7. Latest consensus state: current_term: 1 leader_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } } }
I20260806 04:44:11.398947 24006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:44:11.403118 23973 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } } }
I20260806 04:44:11.403245 23973 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:11.403836 24022 mvcc.cc:205] Tried to move back new op lower bound from 7315420984875495424 to 7315420984774610944. Current Snapshot: MvccSnapshot[applied={T|T < 7315420984875495424}]
I20260806 04:44:11.403776 23974 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } } }
I20260806 04:44:11.403919 23974 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:44:11.404688 24023 mvcc.cc:205] Tried to move back new op lower bound from 7315420984875495424 to 7315420984774610944. Current Snapshot: MvccSnapshot[applied={T|T < 7315420984875495424}]
I20260806 04:44:11.406567 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.371242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55452 (local address 127.31.166.189:36889)
0806 04:44:11.371347 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.371351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.378623 (+ 7272us) server_negotiation.cc:407] Connection header received
0806 04:44:11.378792 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.378795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.378845 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.378905 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.391203 (+ 12298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.391979 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.393156 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.393465 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.393551 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.395948 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:11.395951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:11.395954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:11.395983 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:11.397737 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.404825 (+ 7088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.404829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.404831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.404955 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.405275 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.405279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.405281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.405558 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:11.405719 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:11.406184 (+ 465us) server_negotiation.cc:299] Negotiation successful
0806 04:44:11.406334 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:44:11.410842 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.370680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.31.166.190:39357)
0806 04:44:11.370975 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.370980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.378096 (+ 7116us) server_negotiation.cc:407] Connection header received
0806 04:44:11.378294 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.378297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.378354 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.378414 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.399883 (+ 21469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.400601 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.401710 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.402009 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.402087 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.405401 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:11.405406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:11.405409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:11.405437 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:11.407388 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.409030 (+ 1642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.409035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.409037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.409134 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.409578 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.409585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.409586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.409966 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:11.410114 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:11.410555 (+ 441us) server_negotiation.cc:299] Negotiation successful
0806 04:44:11.410687 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260806 04:44:11.411057 24002 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } } }
I20260806 04:44:11.411221 24002 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7 [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:11.411613 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:11.371832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50314 (local address 127.31.166.188:38507)
0806 04:44:11.371946 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.371950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.379393 (+ 7443us) server_negotiation.cc:407] Connection header received
0806 04:44:11.379582 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.379585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.379633 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.379689 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.380526 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.381364 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.382581 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:11.382964 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:11.383043 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:11.407576 (+ 24533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:11.407581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:11.407583 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:11.407614 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:11.409467 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.410127 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.410133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.410135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.410233 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:11.410555 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:11.410558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:11.410561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:11.410789 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:11.410894 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:11.411331 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 04:44:11.411475 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 04:44:11.412676 24006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8b5ec6069d449ac8d272eb702b8b5e7" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 39357 } } peers { permanent_uuid: "3a64e02b164a435c8a81dd2eeb6eb59e" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 36889 } } peers { permanent_uuid: "e5368d3c60f04db498e809dba30d328d" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 38507 } } }
I20260806 04:44:11.412804 24006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:44:11.414489 24004 catalog_manager.cc:1347] Generated new cluster ID: b28285636c43424ab3fd0145e0943d91
I20260806 04:44:11.414556 24004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:44:11.452112 24004 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:44:11.453423 24004 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:44:11.467438 24004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8b5ec6069d449ac8d272eb702b8b5e7: Generated new TSK 0
I20260806 04:44:11.468390 24004 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:44:11.503178 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:11.499658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.31.166.189:36889)
0806 04:44:11.499800 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.499805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.501762 (+ 1957us) server_negotiation.cc:407] Connection header received
0806 04:44:11.502194 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.502198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.502260 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.502326 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.502998 (+ 672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55462: BlockingRecv error: recv got EOF from 127.0.0.1:55462 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 04:44:11.504302 23966 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:11.500396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50328 (local address 127.31.166.188:38507)
0806 04:44:11.500511 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.500516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.502107 (+ 1591us) server_negotiation.cc:407] Connection header received
0806 04:44:11.503533 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.503537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.503587 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.503638 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.504173 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50328: BlockingRecv error: recv got EOF from 127.0.0.1:50328 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 04:44:11.505831 23806 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:11.501158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.31.166.190:39357)
0806 04:44:11.501272 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.501277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.504787 (+ 3510us) server_negotiation.cc:407] Connection header received
0806 04:44:11.505132 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.505135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.505184 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.505234 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.505700 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59696: BlockingRecv error: recv got EOF from 127.0.0.1:59696 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 04:44:11.510385 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:11.508893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.31.166.189:36889)
0806 04:44:11.508999 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.509003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.509422 (+ 419us) server_negotiation.cc:407] Connection header received
0806 04:44:11.509642 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.509646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.509720 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.509772 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.510283 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55478: BlockingRecv error: recv got EOF from 127.0.0.1:55478 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260806 04:44:11.515328 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:11.513801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55488 (local address 127.31.166.189:36889)
0806 04:44:11.513898 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.513902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.514390 (+ 488us) server_negotiation.cc:407] Connection header received
0806 04:44:11.514560 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.514563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.514609 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.514657 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.515229 (+ 572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55488: BlockingRecv error: recv got EOF from 127.0.0.1:55488 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260806 04:44:11.520681 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:11.519238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55496 (local address 127.31.166.189:36889)
0806 04:44:11.519335 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.519339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.519839 (+ 500us) server_negotiation.cc:407] Connection header received
0806 04:44:11.520018 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.520021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.520073 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.520122 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.520586 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55496: BlockingRecv error: recv got EOF from 127.0.0.1:55496 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260806 04:44:11.528321 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:11.527045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55502 (local address 127.31.166.189:36889)
0806 04:44:11.527221 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.527228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.527431 (+ 203us) server_negotiation.cc:407] Connection header received
0806 04:44:11.527613 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.527617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.527663 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.527713 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.528216 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55502: BlockingRecv error: recv got EOF from 127.0.0.1:55502 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260806 04:44:11.540277 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:11.538577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55508 (local address 127.31.166.189:36889)
0806 04:44:11.538690 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.538694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.539296 (+ 602us) server_negotiation.cc:407] Connection header received
0806 04:44:11.539500 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.539504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.539555 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.539607 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.540163 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55508: BlockingRecv error: recv got EOF from 127.0.0.1:55508 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260806 04:44:11.560570 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:11.558667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.31.166.189:36889)
0806 04:44:11.558808 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.558812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.559477 (+ 665us) server_negotiation.cc:407] Connection header received
0806 04:44:11.559697 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.559701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.559760 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.559834 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.560440 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55520: BlockingRecv error: recv got EOF from 127.0.0.1:55520 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 04:44:11.598089 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:11.595967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.31.166.189:36889)
0806 04:44:11.596089 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.596094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.597079 (+ 985us) server_negotiation.cc:407] Connection header received
0806 04:44:11.597316 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.597320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.597384 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.597478 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.597967 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55536: BlockingRecv error: recv got EOF from 127.0.0.1:55536 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 04:44:11.665490 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:11.664171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.31.166.189:36889)
0806 04:44:11.664352 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.664357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.664519 (+ 162us) server_negotiation.cc:407] Connection header received
0806 04:44:11.664705 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.664708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.664756 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.664856 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.665334 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55542: BlockingRecv error: recv got EOF from 127.0.0.1:55542 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260806 04:44:11.797027 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:11.795917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.31.166.189:36889)
0806 04:44:11.796078 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:11.796082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:11.796221 (+ 139us) server_negotiation.cc:407] Connection header received
0806 04:44:11.796301 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:11.796304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:11.796358 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:11.796444 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:11.796877 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55554: BlockingRecv error: recv got EOF from 127.0.0.1:55554 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 04:44:12.057006 23889 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:12.055806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.31.166.189:36889)
0806 04:44:12.055966 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:12.055970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:12.056064 (+ 94us) server_negotiation.cc:407] Connection header received
0806 04:44:12.056211 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:12.056214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:12.056262 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:12.056336 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:12.056882 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55556: BlockingRecv error: recv got EOF from 127.0.0.1:55556 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 04:44:12.366994 23996 catalog_manager.cc:1259] Loaded cluster ID: b28285636c43424ab3fd0145e0943d91
I20260806 04:44:12.367115 23996 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d: loading cluster ID for follower catalog manager: success
I20260806 04:44:12.369138 23996 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d: acquiring CA information for follower catalog manager: success
I20260806 04:44:12.370311 23996 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5368d3c60f04db498e809dba30d328d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:44:12.379518 24017 catalog_manager.cc:1259] Loaded cluster ID: b28285636c43424ab3fd0145e0943d91
I20260806 04:44:12.379581 24017 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e: loading cluster ID for follower catalog manager: success
I20260806 04:44:12.381161 24017 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e: acquiring CA information for follower catalog manager: success
I20260806 04:44:12.382218 24017 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3a64e02b164a435c8a81dd2eeb6eb59e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:44:12.575901 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:12.571280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.31.166.189:36889)
0806 04:44:12.571577 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:12.571582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:12.571703 (+ 121us) server_negotiation.cc:407] Connection header received
0806 04:44:12.571906 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:12.571908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:12.572036 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:12.572161 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:44:12.572504 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:12.573035 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:12.573936 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:12.574170 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:12.574288 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:12.574687 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:44:12.575385 (+ 698us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:44:12.575524 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:12.575652 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 04:44:12.575708 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260806 04:44:12.582746 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:12.579204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.31.166.188:38507)
0806 04:44:12.579438 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:12.579442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:12.579455 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:44:12.579533 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:12.579535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:12.579657 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:12.579759 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:44:12.579922 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:12.580488 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:12.581362 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:12.581579 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:12.581661 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:12.581998 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:44:12.582411 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:44:12.582506 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:12.582580 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 04:44:12.582626 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":98,"threads_started":1}
I20260806 04:44:12.589957 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:12.586288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50348 (local address 127.31.166.188:38507)
0806 04:44:12.586558 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:12.586565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:12.586579 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:44:12.586649 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:12.586652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:12.586695 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:12.586766 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:44:12.587194 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:12.587966 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:12.588891 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:12.589115 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:12.589230 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:12.589491 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:44:12.589573 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:44:12.589649 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:12.589742 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 04:44:12.589802 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 04:44:12.590484 24116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:12.585809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59698 (local address 127.31.166.190:39357)
0806 04:44:12.586172 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:12.586176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:12.586193 (+ 17us) server_negotiation.cc:407] Connection header received
0806 04:44:12.586387 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:12.586391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:12.586557 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:12.586659 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:44:12.587153 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:12.587737 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:12.588905 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:12.589248 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:12.589342 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:12.589667 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:44:12.590047 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:44:12.590143 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:12.590287 (+ 144us) server_negotiation.cc:299] Negotiation successful
0806 04:44:12.590362 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":69,"threads_started":1}
I20260806 04:44:12.590487 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:12.585859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55580 (local address 127.31.166.189:36889)
0806 04:44:12.585990 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:12.585993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:12.586430 (+ 437us) server_negotiation.cc:407] Connection header received
0806 04:44:12.586632 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:12.586635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:12.586687 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:12.586736 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:44:12.587214 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:12.587990 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:12.589102 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:12.589457 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:12.589532 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:12.589934 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:44:12.590019 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:44:12.590068 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:12.590294 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 04:44:12.590356 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 04:44:12.594054 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 23731
I20260806 04:44:12.601863 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 23809
I20260806 04:44:12.608680 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 23890
2026-08-06T04:44:12Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3119 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 04:44:12 dist-test-slave-vkjp krb5kdc[24120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 04:44:12 dist-test-slave-vkjp krb5kdc[24120](info): set up 2 sockets
Aug 06 04:44:12 dist-test-slave-vkjp krb5kdc[24120](info): commencing operation
krb5kdc: starting...
W20260806 04:44:13.690433 32410 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/test-user.keytab.
Aug 06 04:44:13 dist-test-slave-vkjp krb5kdc[24120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T04:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T04:44:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.166.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
Entry for principal kudu/127.31.166.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab.
I20260806 04:44:13.817970 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:40581
--webserver_interface=127.31.166.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:40581,127.31.166.189:43629,127.31.166.188:33477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 04:44:13.926990 24134 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:44:13.927425 24134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:44:13.927498 24134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:44:13.927578 24134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:44:13.931763 24134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:44:13.931874 24134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:44:13.931914 24134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:44:13.931952 24134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:44:13.931985 24134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:44:13.937513 24134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:40581,127.31.166.189:43629,127.31.166.188:33477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.190.keytab
--principal=kudu/127.31.166.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.190:40581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.166.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:44:13.938681 24134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:44:13.939855 24134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:44:13.948796 24143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:13.948907 24140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:13.948875 24134 server_base.cc:1034] running on GCE node
W20260806 04:44:13.948884 24141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:13.949746 24134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:44:13.950901 24134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:44:13.952123 24134 hybrid_clock.cc:648] HybridClock initialized: now 1785991453952096 us; error 46 us; skew 500 ppm
Aug 06 04:44:13 dist-test-slave-vkjp krb5kdc[24120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991453, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:44:13.955022 24134 init.cc:377] Logged in from keytab as kudu/127.31.166.190@KRBTEST.COM (short username kudu)
I20260806 04:44:13.956140 24134 webserver.cc:460] Webserver started at http://127.31.166.190:36671/ using document root <none> and password file <none>
I20260806 04:44:13.956764 24134 fs_manager.cc:359] Metadata directory not provided
I20260806 04:44:13.956881 24134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:44:13.957139 24134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:44:13.959444 24134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/data/instance:
uuid: "eade0cee9325464eabe8466785859730"
format_stamp: "Formatted at 2026-08-06 04:44:13 on dist-test-slave-vkjp"
server_key: "9ee35494838252eacb8c0ca5ffe68d58"
server_key_iv: "7124a56bbb972be4f8cff36257b60745"
server_key_version: "encryptionkey@0"
I20260806 04:44:13.960062 24134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/wal/instance:
uuid: "eade0cee9325464eabe8466785859730"
format_stamp: "Formatted at 2026-08-06 04:44:13 on dist-test-slave-vkjp"
server_key: "9ee35494838252eacb8c0ca5ffe68d58"
server_key_iv: "7124a56bbb972be4f8cff36257b60745"
server_key_version: "encryptionkey@0"
I20260806 04:44:13.964053 24134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260806 04:44:13.966627 24151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:44:13.967955 24134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:44:13.968103 24134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/wal
uuid: "eade0cee9325464eabe8466785859730"
format_stamp: "Formatted at 2026-08-06 04:44:13 on dist-test-slave-vkjp"
server_key: "9ee35494838252eacb8c0ca5ffe68d58"
server_key_iv: "7124a56bbb972be4f8cff36257b60745"
server_key_version: "encryptionkey@0"
I20260806 04:44:13.968240 24134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:44:13.977988 24134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:44:13.981056 24134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:44:13.981268 24134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:44:13.988517 24134 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.190:40581
I20260806 04:44:13.988819 24205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.190:40581 every 8 connection(s)
I20260806 04:44:13.989564 24134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/data/info.pb
I20260806 04:44:13.992237 24206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:44:13.994544 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 24134
I20260806 04:44:13.994659 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-0/wal/instance
I20260806 04:44:13.994988 32410 external_mini_cluster.cc:1428] Setting key b4c97ebea9a878c0e1a6268fd5cca772
I20260806 04:44:13.996147 24206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.166.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.189@KRBTEST.COM" created.
Aug 06 04:44:14 dist-test-slave-vkjp krb5kdc[24120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991453, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:44:14.013125 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:13.998282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40581 (local address 127.0.0.1:34590)
0806 04:44:13.998831 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:13.998893 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:14.000186 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:14.001001 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:14.001007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:14.001503 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:14.002013 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.002039 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.003318 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.003322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.004381 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.004391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.005156 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.005159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.005255 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.005819 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:14.005861 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:14.007768 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:14.010052 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.010062 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.010070 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.010362 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.010760 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.010762 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.010764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.010881 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.011744 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:14.011750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:14.011884 (+ 134us) client_negotiation.cc:769] Sending connection context
0806 04:44:14.012033 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 04:44:14.012297 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":105,"threads_started":1}
I20260806 04:44:14.013128 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:13.998987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.31.166.190:40581)
0806 04:44:13.999285 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:13.999294 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:13.999331 (+ 37us) server_negotiation.cc:407] Connection header received
0806 04:44:14.000538 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.000543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.000758 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.000932 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.002207 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.003188 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.004501 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.004977 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.005103 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.007922 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.007929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.007947 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.007968 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.009933 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.010484 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.010493 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.010498 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.010653 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.010989 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.010995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.010999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.011614 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.011763 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.012171 (+ 408us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.012383 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":127,"threads_started":1}
I20260806 04:44:14.015987 24206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:44:14.016891 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.016510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:43629 (local address 127.0.0.1:37266)
0806 04:44:14.016727 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.016822 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260806 04:44:14.017091 24155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.166.189:43629: connect: Connection refused (error 111)
I20260806 04:44:14.018085 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.017823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:43629 (local address 127.0.0.1:37268)
0806 04:44:14.017977 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.018032 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260806 04:44:14.018224 24206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.189:43629: Network error: Client connection negotiation failed: client connection to 127.31.166.189:43629: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.189.keytab.
Entry for principal kudu/127.31.166.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.189.keytab.
I20260806 04:44:14.026163 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.189:43629
--webserver_interface=127.31.166.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.189.keytab
--principal=kudu/127.31.166.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:40581,127.31.166.189:43629,127.31.166.188:33477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:44:14.067466 24206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } attempt: 1
I20260806 04:44:14.068423 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.068110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:43629 (local address 127.0.0.1:37284)
0806 04:44:14.068274 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.068338 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 04:44:14.069149 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.068933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:43629 (local address 127.0.0.1:37290)
0806 04:44:14.069043 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.069096 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 04:44:14.069262 24206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.189:43629: Network error: Client connection negotiation failed: client connection to 127.31.166.189:43629: connect: Connection refused (error 111)
I20260806 04:44:14.137444 24206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } attempt: 2
I20260806 04:44:14.138370 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.138031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:43629 (local address 127.0.0.1:37306)
0806 04:44:14.138206 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.138275 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 04:44:14.139223 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.138972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:43629 (local address 127.0.0.1:37318)
0806 04:44:14.139072 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.139170 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.189:43629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260806 04:44:14.139436 24206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.189:43629: Network error: Client connection negotiation failed: client connection to 127.31.166.189:43629: connect: Connection refused (error 111)
W20260806 04:44:14.141469 24213 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:44:14.141765 24213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:44:14.141825 24213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:44:14.141904 24213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:44:14.146023 24213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:44:14.146098 24213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:44:14.146164 24213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:44:14.146219 24213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:44:14.146256 24213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:44:14.151520 24213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:40581,127.31.166.189:43629,127.31.166.188:33477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.189.keytab
--principal=kudu/127.31.166.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.189:43629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.166.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:44:14.152592 24213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:44:14.153594 24213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:44:14.161830 24222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:14.162144 24213 server_base.cc:1034] running on GCE node
W20260806 04:44:14.161865 24219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:14.161866 24220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:14.162892 24213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:44:14.164093 24213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:44:14.165285 24213 hybrid_clock.cc:648] HybridClock initialized: now 1785991454165266 us; error 31 us; skew 500 ppm
Aug 06 04:44:14 dist-test-slave-vkjp krb5kdc[24120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991454, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:44:14.172792 24213 init.cc:377] Logged in from keytab as kudu/127.31.166.189@KRBTEST.COM (short username kudu)
I20260806 04:44:14.173748 24213 webserver.cc:460] Webserver started at http://127.31.166.189:45145/ using document root <none> and password file <none>
I20260806 04:44:14.174358 24213 fs_manager.cc:359] Metadata directory not provided
I20260806 04:44:14.174458 24213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:44:14.174691 24213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:44:14.176853 24213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/data/instance:
uuid: "fa435521895a4fbabd17557a22e8c8bc"
format_stamp: "Formatted at 2026-08-06 04:44:14 on dist-test-slave-vkjp"
server_key: "d4e9c061c1bd98c9678545ce99d898fd"
server_key_iv: "02b6acf8a0e781e4f5d7a7e62b44a2d4"
server_key_version: "encryptionkey@0"
I20260806 04:44:14.177419 24213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/wal/instance:
uuid: "fa435521895a4fbabd17557a22e8c8bc"
format_stamp: "Formatted at 2026-08-06 04:44:14 on dist-test-slave-vkjp"
server_key: "d4e9c061c1bd98c9678545ce99d898fd"
server_key_iv: "02b6acf8a0e781e4f5d7a7e62b44a2d4"
server_key_version: "encryptionkey@0"
I20260806 04:44:14.181262 24213 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 04:44:14.183802 24230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:44:14.185022 24213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260806 04:44:14.185171 24213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/wal
uuid: "fa435521895a4fbabd17557a22e8c8bc"
format_stamp: "Formatted at 2026-08-06 04:44:14 on dist-test-slave-vkjp"
server_key: "d4e9c061c1bd98c9678545ce99d898fd"
server_key_iv: "02b6acf8a0e781e4f5d7a7e62b44a2d4"
server_key_version: "encryptionkey@0"
I20260806 04:44:14.185285 24213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:44:14.198078 24213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:44:14.200976 24213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:44:14.201201 24213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:44:14.208650 24213 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.189:43629
I20260806 04:44:14.208957 24285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.189:43629 every 8 connection(s)
I20260806 04:44:14.209664 24213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/data/info.pb
I20260806 04:44:14.212428 24286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:44:14.213029 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 24213
I20260806 04:44:14.213208 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-1/wal/instance
I20260806 04:44:14.213546 32410 external_mini_cluster.cc:1428] Setting key fec3ea4beb97b2e34daf6fe4b3f2b2d7
I20260806 04:44:14.216936 24286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.166.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.166.188@KRBTEST.COM" created.
Aug 06 04:44:14 dist-test-slave-vkjp krb5kdc[24120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991454, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.189@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:44:14.230600 24206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } attempt: 3
I20260806 04:44:14.232612 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.219434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.31.166.190:40581)
0806 04:44:14.219608 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.219613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.220475 (+ 862us) server_negotiation.cc:407] Connection header received
0806 04:44:14.221431 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.221436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.221507 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.221618 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.222850 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.223579 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.224680 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.225077 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.225174 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.228336 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.228339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.228341 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.228375 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.229854 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.230419 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.230422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.230424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.230504 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.231052 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.231057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.231061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.231380 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.231491 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.232231 (+ 740us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.232384 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 04:44:14.234261 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.219822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40581 (local address 127.0.0.1:34592)
0806 04:44:14.220290 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.220342 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:14.221217 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:14.221769 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:14.221775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:14.222174 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:14.222687 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.222700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.223706 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.223709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.224553 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.224562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.225251 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.225258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.225496 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.226146 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:14.226173 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:14.228156 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:14.229966 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.229972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.229983 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.230266 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.230655 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.230665 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.230667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.230931 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.231555 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:14.231562 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:14.231905 (+ 343us) client_negotiation.cc:769] Sending connection context
0806 04:44:14.232776 (+ 871us) client_negotiation.cc:240] Negotiation successful
0806 04:44:14.233146 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":136,"threads_started":1}
I20260806 04:44:14.235791 24286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:44:14 dist-test-slave-vkjp krb5kdc[24120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991453, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for kudu/127.31.166.189@KRBTEST.COM
Aug 06 04:44:14 dist-test-slave-vkjp krb5kdc[24120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991454, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.189@KRBTEST.COM for kudu/127.31.166.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.166.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.188.keytab.
Entry for principal kudu/127.31.166.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.188.keytab.
I20260806 04:44:14.241791 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.231147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:43629 (local address 127.0.0.1:37330)
0806 04:44:14.231341 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.231368 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:14.231454 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:14.232122 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:14.232125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:14.232323 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:14.232413 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.232418 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.233649 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.233655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.234198 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.234206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.234965 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.234969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.235072 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.235529 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:14.235543 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:14.237787 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:14.239961 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.239964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.239966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.240234 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.240547 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.240551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.240553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.240643 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.241282 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:14.241285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:14.241379 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 04:44:14.241470 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 04:44:14.241615 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260806 04:44:14.242684 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.231249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.31.166.189:43629)
0806 04:44:14.231754 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.231768 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.231794 (+ 26us) server_negotiation.cc:407] Connection header received
0806 04:44:14.231836 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.231840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.231992 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.232138 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.232567 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.233520 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.234327 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.234863 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.234968 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.237933 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.237943 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.237956 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.237987 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.239834 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.240368 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.240373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.240374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.240450 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.240767 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.240771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.240773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.241108 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.241314 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.241583 (+ 269us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.241768 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":164,"threads_started":1}
I20260806 04:44:14.243388 32410 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskk51eOb/build/debug/bin/kudu
/tmp/dist-test-taskk51eOb/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.188:33477
--webserver_interface=127.31.166.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.188.keytab
--principal=kudu/127.31.166.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.166.148:46859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.166.190:40581,127.31.166.189:43629,127.31.166.188:33477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 04:44:14.244874 24206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:44:14.245136 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.236352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:43629 (local address 127.0.0.1:37336)
0806 04:44:14.236524 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.236540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:14.236611 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:14.236960 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:14.236963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:14.237125 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:14.237199 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.237204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.238012 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.238016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.238741 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.238754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.239446 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.239450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.239547 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.240025 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:14.240037 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:14.241482 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:14.243643 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.243648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.243650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.243863 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.244148 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.244153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.244155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.244237 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.244665 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:14.244668 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:14.244754 (+ 86us) client_negotiation.cc:769] Sending connection context
0806 04:44:14.244850 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 04:44:14.244972 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260806 04:44:14.245640 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.236398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.31.166.189:43629)
0806 04:44:14.236727 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.236738 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.236754 (+ 16us) server_negotiation.cc:407] Connection header received
0806 04:44:14.236795 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.236800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.236840 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.236951 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.237306 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.237899 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.238906 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.239293 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.239414 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.241675 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.241694 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.241697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.241721 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.243528 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.243964 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.243968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.243970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.244059 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.244335 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.244338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.244340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.244547 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.244632 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.245358 (+ 726us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.245474 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":117,"threads_started":1}
I20260806 04:44:14.245779 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.245499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47664)
0806 04:44:14.245644 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.245712 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 04:44:14.246511 24286 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:44:14.246639 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.246389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47666)
0806 04:44:14.246536 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.246584 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260806 04:44:14.246805 24206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:33477: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
I20260806 04:44:14.247211 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.246901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47670)
0806 04:44:14.247019 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.247151 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 04:44:14.247395 24231 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
I20260806 04:44:14.248167 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.247963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47680)
0806 04:44:14.248068 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.248112 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:44:14.248323 24286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:33477: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
I20260806 04:44:14.277966 24206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } attempt: 1
I20260806 04:44:14.278901 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.278592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47686)
0806 04:44:14.278750 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.278814 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 04:44:14.279695 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.279471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47694)
0806 04:44:14.279586 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.279640 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 04:44:14.279834 24206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:33477: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
I20260806 04:44:14.297581 24286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } attempt: 1
I20260806 04:44:14.298629 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.298271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47700)
0806 04:44:14.298454 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.298538 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 04:44:14.299547 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.299289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47702)
0806 04:44:14.299444 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.299490 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 04:44:14.299721 24286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:33477: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
I20260806 04:44:14.355012 24206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } attempt: 2
I20260806 04:44:14.355933 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.355609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47704)
0806 04:44:14.355757 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.355833 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 04:44:14.356784 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.356541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47712)
0806 04:44:14.356673 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.356728 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 04:44:14.356962 24206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:33477: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
W20260806 04:44:14.359885 24293 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 04:44:14.360183 24293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 04:44:14.360232 24293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 04:44:14.360323 24293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 04:44:14.364475 24293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 04:44:14.364527 24293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 04:44:14.364548 24293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 04:44:14.364585 24293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 04:44:14.364616 24293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 04:44:14.367883 24286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } attempt: 2
I20260806 04:44:14.368745 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.368451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47716)
0806 04:44:14.368599 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.368670 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 04:44:14.369505 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.369306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47724)
0806 04:44:14.369411 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.369457 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 04:44:14.369643 24286 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.166.188:33477: Network error: Client connection negotiation failed: client connection to 127.31.166.188:33477: connect: Connection refused (error 111)
I20260806 04:44:14.370864 24293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.166.148:46859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.166.190:40581,127.31.166.189:43629,127.31.166.188:33477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/krb5kdc/kudu_127.31.166.188.keytab
--principal=kudu/127.31.166.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.166.188:33477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.166.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 06 Aug 2026 03:57:55 UTC on ccycloud.quasar-jhkjat.root.comops.site
build id 4569616
I20260806 04:44:14.372123 24293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 04:44:14.373224 24293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 04:44:14.381816 24302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:14.381789 24300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 04:44:14.381780 24299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 04:44:14.382203 24293 server_base.cc:1034] running on GCE node
I20260806 04:44:14.382951 24293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 04:44:14.384073 24293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 04:44:14.385246 24293 hybrid_clock.cc:648] HybridClock initialized: now 1785991454385234 us; error 32 us; skew 500 ppm
Aug 06 04:44:14 dist-test-slave-vkjp krb5kdc[24120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991454, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 04:44:14.388212 24293 init.cc:377] Logged in from keytab as kudu/127.31.166.188@KRBTEST.COM (short username kudu)
I20260806 04:44:14.389225 24293 webserver.cc:460] Webserver started at http://127.31.166.188:43267/ using document root <none> and password file <none>
I20260806 04:44:14.389791 24293 fs_manager.cc:359] Metadata directory not provided
I20260806 04:44:14.389863 24293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 04:44:14.390082 24293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 04:44:14.392275 24293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/data/instance:
uuid: "af78723eb85c47aeafeb85fcc2ef3507"
format_stamp: "Formatted at 2026-08-06 04:44:14 on dist-test-slave-vkjp"
server_key: "8ddd48604fa57a1f2476b9970c8f0894"
server_key_iv: "4adb63a69c9ea742d251c6261d6bcaac"
server_key_version: "encryptionkey@0"
I20260806 04:44:14.392946 24293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/wal/instance:
uuid: "af78723eb85c47aeafeb85fcc2ef3507"
format_stamp: "Formatted at 2026-08-06 04:44:14 on dist-test-slave-vkjp"
server_key: "8ddd48604fa57a1f2476b9970c8f0894"
server_key_iv: "4adb63a69c9ea742d251c6261d6bcaac"
server_key_version: "encryptionkey@0"
I20260806 04:44:14.396816 24293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 04:44:14.399413 24309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 04:44:14.400701 24293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 04:44:14.400851 24293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/data,/tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/wal
uuid: "af78723eb85c47aeafeb85fcc2ef3507"
format_stamp: "Formatted at 2026-08-06 04:44:14 on dist-test-slave-vkjp"
server_key: "8ddd48604fa57a1f2476b9970c8f0894"
server_key_iv: "4adb63a69c9ea742d251c6261d6bcaac"
server_key_version: "encryptionkey@0"
I20260806 04:44:14.400970 24293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 04:44:14.423266 24293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 04:44:14.426395 24293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 04:44:14.426612 24293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 04:44:14.434170 24293 rpc_server.cc:307] RPC server started. Bound to: 127.31.166.188:33477
I20260806 04:44:14.434504 24366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.166.188:33477 every 8 connection(s)
I20260806 04:44:14.435194 24293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/data/info.pb
I20260806 04:44:14.437881 24367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 04:44:14.441255 32410 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskk51eOb/build/debug/bin/kudu as pid 24293
I20260806 04:44:14.441408 32410 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskk51eOb/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785991160137184-32410-0/minicluster-data/master-2/wal/instance
I20260806 04:44:14.441749 32410 external_mini_cluster.cc:1428] Setting key a7f7624a658f50350e5c93bd26a522be
I20260806 04:44:14.442701 24367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 04:44:14 dist-test-slave-vkjp krb5kdc[24120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.188@KRBTEST.COM
Aug 06 04:44:14 dist-test-slave-vkjp krb5kdc[24120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991454, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.188@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:44:14.455530 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.444679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.31.166.190:40581)
0806 04:44:14.444832 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.444836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.445132 (+ 296us) server_negotiation.cc:407] Connection header received
0806 04:44:14.445327 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.445330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.445378 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.445428 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.446526 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.447024 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.448075 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.448341 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.448429 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.451943 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.451946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.451948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.451973 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.453697 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.454197 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.454199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.454200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.454273 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.454541 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.454543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.454544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.454758 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.454859 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.455280 (+ 421us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.455410 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 04:44:14.456110 24206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } attempt: 3
I20260806 04:44:14.456473 24371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.444602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.190:40581 (local address 127.0.0.1:34596)
0806 04:44:14.445003 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.445056 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:14.445214 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:14.445620 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:14.445625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:14.445980 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:14.446377 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.446391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.447166 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.447169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.447951 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.447960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.448488 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.448492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.448687 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.449545 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:14.449562 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:14.451772 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:14.453797 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.453807 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.453814 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.454102 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.454359 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.454362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.454364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.454450 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.454868 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:14.454873 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:14.455030 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 04:44:14.455183 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 04:44:14.455454 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":45,"spinlock_wait_cycles":896,"thread_start_us":150,"threads_started":1}
I20260806 04:44:14.456473 24370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.443190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47730 (local address 127.31.166.188:33477)
0806 04:44:14.443593 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.443612 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.443653 (+ 41us) server_negotiation.cc:407] Connection header received
0806 04:44:14.444432 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.444450 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.444879 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.445221 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.446103 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.447257 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.448111 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.448526 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.448695 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.451262 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.451267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.451276 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.451327 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.453162 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.453629 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.453633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.453638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.453759 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.454047 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.454051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.454053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.454597 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.454757 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.455040 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.455517 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":165,"server-negotiator.queue_time_us":266,"spinlock_wait_cycles":1280,"thread_start_us":109,"threads_started":1}
I20260806 04:44:14.457768 24367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:44:14.460796 24286 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } attempt: 3
Aug 06 04:44:14 dist-test-slave-vkjp krb5kdc[24120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991453, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.190@KRBTEST.COM for kudu/127.31.166.188@KRBTEST.COM
Aug 06 04:44:14 dist-test-slave-vkjp krb5kdc[24120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991454, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.188@KRBTEST.COM for kudu/127.31.166.189@KRBTEST.COM
Aug 06 04:44:14 dist-test-slave-vkjp krb5kdc[24120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.190@KRBTEST.COM
I20260806 04:44:14.468429 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.458487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.31.166.189:43629)
0806 04:44:14.458637 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.458641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.458656 (+ 15us) server_negotiation.cc:407] Connection header received
0806 04:44:14.458767 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.458770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.458823 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.458926 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.459754 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.460514 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.461529 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.461816 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.461873 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.464447 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.464453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.464455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.464480 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.466172 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.466711 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.466717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.466719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.466810 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.467162 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.467169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.467422 (+ 253us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.467674 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.467793 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.468126 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.468257 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 04:44:14.468567 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.456529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47744)
0806 04:44:14.456630 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.456647 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:14.456722 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:14.457561 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:14.457564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:14.457734 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:14.457835 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.457840 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.459109 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.459114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.459927 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.459936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.460519 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.460522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.460597 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.461126 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:14.461140 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:14.462808 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:14.466020 (+ 3212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.466024 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.466027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.467070 (+ 1043us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.467430 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.467432 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.467434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.467533 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.468085 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:14.468088 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:14.468179 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 04:44:14.468265 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 04:44:14.468396 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":1}
Aug 06 04:44:14 dist-test-slave-vkjp krb5kdc[24120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991454, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.189@KRBTEST.COM for kudu/127.31.166.188@KRBTEST.COM
I20260806 04:44:14.468910 24371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.458232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.189:43629 (local address 127.0.0.1:37344)
0806 04:44:14.458577 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.458591 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:14.458661 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:14.459338 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:14.459341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:14.459508 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:14.459612 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.459618 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.460630 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.460633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.461405 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.461416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.461992 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.461994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.462067 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.462506 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:14.462524 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:14.464279 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:14.466321 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.466326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.466329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.466586 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.466932 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.466936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.466938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.467034 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.467827 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:14.467832 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:14.467926 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 04:44:14.468616 (+ 690us) client_negotiation.cc:240] Negotiation successful
0806 04:44:14.468747 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"spinlock_wait_cycles":768}
I20260806 04:44:14.469517 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.456722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47744 (local address 127.31.166.188:33477)
0806 04:44:14.457317 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.457327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.457340 (+ 13us) server_negotiation.cc:407] Connection header received
0806 04:44:14.457382 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.457385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.457427 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.457967 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.458078 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.458993 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.460040 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.460388 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.460519 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.462918 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.462923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.462927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.462950 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.465886 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.467224 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.467228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.467232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.467328 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.467777 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.467781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.467783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.467964 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.469127 (+ 1163us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.469199 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.469313 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":155,"threads_started":1}
I20260806 04:44:14.469573 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.458962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.31.166.190:40581)
0806 04:44:14.459071 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.459075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.459086 (+ 11us) server_negotiation.cc:407] Connection header received
0806 04:44:14.459134 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.459136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.459173 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.459245 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.459812 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.460373 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.461488 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.461816 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.461892 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.465185 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.465188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.465190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.465214 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.466951 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.467563 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.467566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.467568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.467649 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.467987 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.467991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.467993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.468241 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.468438 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.469294 (+ 856us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.469413 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 04:44:14.469563 24367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } has no permanent_uuid. Determining permanent_uuid...
I20260806 04:44:14.473340 24206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730: Bootstrap starting.
I20260806 04:44:14.473533 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.461206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47746)
0806 04:44:14.461410 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.461423 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:14.461500 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:14.462800 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:14.462804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:14.463006 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:14.463135 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.463143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.464463 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.464467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.466427 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.466436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.466905 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.466909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.466992 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.467660 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:14.467672 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:14.469243 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:14.471757 (+ 2514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.471760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.471762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.472023 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.472561 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.472564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.472566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.472651 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.473081 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:14.473085 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:14.473176 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 04:44:14.473257 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 04:44:14.473387 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260806 04:44:14.474112 24370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.461776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.31.166.188:33477)
0806 04:44:14.462598 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.462602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.462614 (+ 12us) server_negotiation.cc:407] Connection header received
0806 04:44:14.462648 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.462651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.462690 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.462774 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.463541 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.464354 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.466544 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.466771 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.466896 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.469401 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.469404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.469406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.469428 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.471607 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.472361 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.472364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.472366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.472450 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.472764 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.472766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.472768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.472964 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.473772 (+ 808us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.473859 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.473962 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751}
I20260806 04:44:14.476300 24206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730: Neither blocks nor log segments found. Creating new log.
Aug 06 04:44:14 dist-test-slave-vkjp krb5kdc[24120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991454, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.166.188@KRBTEST.COM for kudu/127.31.166.188@KRBTEST.COM
I20260806 04:44:14.477193 24206 log.cc:824] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730: Log is configured to *not* fsync() on all Append() calls
I20260806 04:44:14.477308 24286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc: Bootstrap starting.
Aug 06 04:44:14 dist-test-slave-vkjp krb5kdc[24120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785991453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.166.189@KRBTEST.COM
I20260806 04:44:14.479465 24206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730: No bootstrap required, opened a new log
I20260806 04:44:14.479991 24286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc: Neither blocks nor log segments found. Creating new log.
I20260806 04:44:14.480751 24371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.470022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.166.188:33477 (local address 127.0.0.1:47748)
0806 04:44:14.470424 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0806 04:44:14.470437 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 04:44:14.470509 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 04:44:14.471975 (+ 1466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 04:44:14.471978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 04:44:14.472138 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0806 04:44:14.472264 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.472271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.473273 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.473276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.474037 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 04:44:14.474046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.474683 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.474686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 04:44:14.474756 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.475186 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 04:44:14.475205 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 04:44:14.476689 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 04:44:14.478781 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.478783 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.478785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.479020 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.479338 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 04:44:14.479343 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 04:44:14.479347 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 04:44:14.479684 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 04:44:14.480171 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 04:44:14.480174 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 04:44:14.480295 (+ 121us) client_negotiation.cc:769] Sending connection context
0806 04:44:14.480411 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 04:44:14.480554 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340}
I20260806 04:44:14.480825 24286 log.cc:824] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc: Log is configured to *not* fsync() on all Append() calls
I20260806 04:44:14.481395 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.470214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.31.166.188:33477)
0806 04:44:14.470309 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.470312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.470756 (+ 444us) server_negotiation.cc:407] Connection header received
0806 04:44:14.470871 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.470874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.471878 (+ 1004us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.471965 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.472392 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.473168 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.474145 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.474497 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.474592 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.476843 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.476847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.476853 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.476883 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.478658 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.479151 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.479154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.479156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.479221 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.479795 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.479801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.479802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.480032 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.480129 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.481034 (+ 905us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.481164 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 04:44:14.483072 24286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc: No bootstrap required, opened a new log
I20260806 04:44:14.483592 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.471549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.31.166.189:43629)
0806 04:44:14.471642 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.471646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.471724 (+ 78us) server_negotiation.cc:407] Connection header received
0806 04:44:14.472140 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.472144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.472195 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.472269 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.472991 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.473733 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.475283 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.475617 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.475710 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.479109 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.479113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.479116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.479144 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.480922 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.481876 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.481880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.481881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.481971 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.482299 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.482302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.482304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.482529 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.482723 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.483296 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.483418 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260806 04:44:14.483752 24206 raft_consensus.cc:348] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } }
I20260806 04:44:14.484014 24206 raft_consensus.cc:729] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eade0cee9325464eabe8466785859730, State: Initialized, Role: FOLLOWER
I20260806 04:44:14.484371 24206 consensus_queue.cc:260] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } }
I20260806 04:44:14.484498 24367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507: Bootstrap starting.
I20260806 04:44:14.485814 24206 sys_catalog.cc:564] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:44:14.487241 24367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507: Neither blocks nor log segments found. Creating new log.
I20260806 04:44:14.487305 24286 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } }
I20260806 04:44:14.487547 24286 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa435521895a4fbabd17557a22e8c8bc, State: Initialized, Role: FOLLOWER
I20260806 04:44:14.487854 32410 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 04:44:14.487952 24286 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } }
I20260806 04:44:14.488251 24367 log.cc:824] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507: Log is configured to *not* fsync() on all Append() calls
I20260806 04:44:14.489228 24180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "eade0cee9325464eabe8466785859730"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34602
I20260806 04:44:14.489451 24180 raft_consensus.cc:482] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 04:44:14.489570 24180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:44:14.488552 24376 sys_catalog.cc:455] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } } }
I20260806 04:44:14.489773 24376 sys_catalog.cc:458] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:44:14.490183 24286 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 04:44:14.490758 24367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507: No bootstrap required, opened a new log
I20260806 04:44:14.490862 24180 raft_consensus.cc:504] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } }
I20260806 04:44:14.491765 24180 leader_election.cc:288] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [CANDIDATE]: Term 1 election: Requested vote from peers fa435521895a4fbabd17557a22e8c8bc (127.31.166.189:43629), af78723eb85c47aeafeb85fcc2ef3507 (127.31.166.188:33477)
I20260806 04:44:14.493152 24257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eade0cee9325464eabe8466785859730" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fa435521895a4fbabd17557a22e8c8bc"
I20260806 04:44:14.493861 24257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 04:44:14.490754 24134 master_runner.cc:186] Error getting master registration for 127.31.166.189:43629: OK, instance_id { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" instance_seqno: 1785991454206648 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 04:44:14.494978 24134 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:44:14.495257 24257 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eade0cee9325464eabe8466785859730 in term 1.
I20260806 04:44:14.495774 24155 leader_election.cc:302] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eade0cee9325464eabe8466785859730, fa435521895a4fbabd17557a22e8c8bc; no voters:
I20260806 04:44:14.495888 24367 raft_consensus.cc:348] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } }
I20260806 04:44:14.496131 24367 raft_consensus.cc:729] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af78723eb85c47aeafeb85fcc2ef3507, State: Initialized, Role: FOLLOWER
I20260806 04:44:14.496456 24367 consensus_queue.cc:260] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } }
I20260806 04:44:14.497700 24367 sys_catalog.cc:564] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 04:44:14.498456 24400 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260806 04:44:14.498399 24337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eade0cee9325464eabe8466785859730" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af78723eb85c47aeafeb85fcc2ef3507"
W20260806 04:44:14.498530 24400 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260806 04:44:14.498458 24387 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260806 04:44:14.498625 24337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507 [term 0 FOLLOWER]: Advancing to term 1
W20260806 04:44:14.498660 24387 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:44:14.498944 24388 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } } }
I20260806 04:44:14.499063 24388 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:44:14.500039 24337 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eade0cee9325464eabe8466785859730 in term 1.
I20260806 04:44:14.502015 24376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 04:44:14.502229 24376 raft_consensus.cc:686] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [term 1 LEADER]: Becoming Leader. State: Replica: eade0cee9325464eabe8466785859730, State: Running, Role: LEADER
I20260806 04:44:14.502569 24376 consensus_queue.cc:237] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } }
I20260806 04:44:14.503809 24213 master_runner.cc:263] No leader master found from master fa435521895a4fbabd17557a22e8c8bc
I20260806 04:44:14.503880 24213 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 04:44:14.507310 24405 sys_catalog.cc:455] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } } }
I20260806 04:44:14.507440 24405 sys_catalog.cc:458] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:44:14.510805 24376 sys_catalog.cc:455] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eade0cee9325464eabe8466785859730. Latest consensus state: current_term: 1 leader_uuid: "eade0cee9325464eabe8466785859730" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } } }
I20260806 04:44:14.510944 24376 sys_catalog.cc:458] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:14.517412 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.499260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.31.166.190:40581)
0806 04:44:14.499431 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.499437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.501348 (+ 1911us) server_negotiation.cc:407] Connection header received
0806 04:44:14.501895 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.501898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.501947 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.501999 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.506241 (+ 4242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.506990 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.510924 (+ 3934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.511289 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.511368 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.513444 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.513449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.513454 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.513481 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.515303 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.515941 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.515946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.515949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.516068 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.516394 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.516400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.516404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.516631 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.516735 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.517096 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.517224 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260806 04:44:14.518621 24419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 04:44:14.518699 24419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 04:44:14.521279 24372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.499832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.31.166.188:33477)
0806 04:44:14.499927 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.499931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.501154 (+ 1223us) server_negotiation.cc:407] Connection header received
0806 04:44:14.501632 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.501635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.501685 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.501737 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.504220 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.504993 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.509756 (+ 4763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.510104 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.510182 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.514962 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.514967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.514970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.514997 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.516697 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.518584 (+ 1887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.518588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.518589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.518687 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.520238 (+ 1551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.520243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.520247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.520473 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.520619 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.520947 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.521086 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260806 04:44:14.523435 24425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 04:44:14.524585 24425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 04:44:14.525372 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:14.500734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.31.166.189:43629)
0806 04:44:14.500855 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.500859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.507741 (+ 6882us) server_negotiation.cc:407] Connection header received
0806 04:44:14.508701 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.508704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.508753 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.508822 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.511977 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.512821 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.517211 (+ 4390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:14.517531 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:14.517614 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:14.521317 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 04:44:14.521322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 04:44:14.521326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 04:44:14.521361 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 04:44:14.523370 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.523934 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.523937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.523939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.524024 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 04:44:14.524339 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 04:44:14.524342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 04:44:14.524344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 04:44:14.524564 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 04:44:14.524695 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:14.525109 (+ 414us) server_negotiation.cc:299] Negotiation successful
0806 04:44:14.525226 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260806 04:44:14.532407 24337 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507 [term 1 FOLLOWER]: Refusing update from remote peer eade0cee9325464eabe8466785859730: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:44:14.532624 24257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc [term 1 FOLLOWER]: Refusing update from remote peer eade0cee9325464eabe8466785859730: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 04:44:14.533063 24423 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:44:14.533308 24376 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 04:44:14.536909 24405 sys_catalog.cc:455] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eade0cee9325464eabe8466785859730. Latest consensus state: current_term: 1 leader_uuid: "eade0cee9325464eabe8466785859730" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } } }
I20260806 04:44:14.537025 24405 sys_catalog.cc:458] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:44:14.538671 24388 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader eade0cee9325464eabe8466785859730. Latest consensus state: current_term: 1 leader_uuid: "eade0cee9325464eabe8466785859730" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } } }
I20260806 04:44:14.538813 24388 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:44:14.541571 24423 sys_catalog.cc:455] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eade0cee9325464eabe8466785859730" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } } }
I20260806 04:44:14.541688 24423 sys_catalog.cc:458] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:14.542659 24388 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eade0cee9325464eabe8466785859730" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } } }
I20260806 04:44:14.542763 24388 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:44:14.543430 24428 mvcc.cc:205] Tried to move back new op lower bound from 7315420997756465152 to 7315420997651595264. Current Snapshot: MvccSnapshot[applied={T|T < 7315420997756465152}]
I20260806 04:44:14.545699 24431 mvcc.cc:205] Tried to move back new op lower bound from 7315420997756465152 to 7315420997651595264. Current Snapshot: MvccSnapshot[applied={T|T < 7315420997756465152}]
I20260806 04:44:14.548045 24425 catalog_manager.cc:1347] Generated new cluster ID: 8fa87a362cb14c8d9b381e9bd246ffd3
I20260806 04:44:14.548118 24425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 04:44:14.553764 24376 sys_catalog.cc:455] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eade0cee9325464eabe8466785859730" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } } }
I20260806 04:44:14.553900 24376 sys_catalog.cc:458] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730 [sys.catalog]: This master's current role is: LEADER
I20260806 04:44:14.555042 24405 sys_catalog.cc:455] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eade0cee9325464eabe8466785859730" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eade0cee9325464eabe8466785859730" member_type: VOTER last_known_addr { host: "127.31.166.190" port: 40581 } } peers { permanent_uuid: "fa435521895a4fbabd17557a22e8c8bc" member_type: VOTER last_known_addr { host: "127.31.166.189" port: 43629 } } peers { permanent_uuid: "af78723eb85c47aeafeb85fcc2ef3507" member_type: VOTER last_known_addr { host: "127.31.166.188" port: 33477 } } }
I20260806 04:44:14.555186 24405 sys_catalog.cc:458] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 04:44:14.555856 24429 mvcc.cc:205] Tried to move back new op lower bound from 7315420997756465152 to 7315420997651595264. Current Snapshot: MvccSnapshot[applied={T|T < 7315420997756465152}]
I20260806 04:44:14.586227 24425 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 04:44:14.588569 24425 catalog_manager.cc:1504] Loading token signing keys...
I20260806 04:44:14.605598 24425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eade0cee9325464eabe8466785859730: Generated new TSK 0
I20260806 04:44:14.616024 24425 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 04:44:14.648710 24210 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:14.645873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.31.166.190:40581)
0806 04:44:14.646015 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.646020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.647380 (+ 1360us) server_negotiation.cc:407] Connection header received
0806 04:44:14.647797 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.647801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.647857 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.647912 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.648567 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34630: BlockingRecv error: recv got EOF from 127.0.0.1:34630 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 04:44:14.649834 24292 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:14.646666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.31.166.189:43629)
0806 04:44:14.646779 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.646783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.647708 (+ 925us) server_negotiation.cc:407] Connection header received
0806 04:44:14.649078 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.649083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.649132 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.649182 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.649722 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37368: BlockingRecv error: recv got EOF from 127.0.0.1:37368 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 04:44:14.651973 24372 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:14.650631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.31.166.188:33477)
0806 04:44:14.650737 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.650741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.651018 (+ 277us) server_negotiation.cc:407] Connection header received
0806 04:44:14.651260 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.651263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.651324 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.651375 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.651855 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47774: BlockingRecv error: recv got EOF from 127.0.0.1:47774 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 04:44:14.656224 24292 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:14.654647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37378 (local address 127.31.166.189:43629)
0806 04:44:14.654744 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.654748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.655267 (+ 519us) server_negotiation.cc:407] Connection header received
0806 04:44:14.655511 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.655514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.655588 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.655648 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.656127 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37378: BlockingRecv error: recv got EOF from 127.0.0.1:37378 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260806 04:44:14.660874 24292 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:14.659453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.31.166.189:43629)
0806 04:44:14.659549 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.659553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.660038 (+ 485us) server_negotiation.cc:407] Connection header received
0806 04:44:14.660208 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.660211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.660257 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.660320 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.660785 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37390: BlockingRecv error: recv got EOF from 127.0.0.1:37390 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260806 04:44:14.666100 24292 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:14.664688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.31.166.189:43629)
0806 04:44:14.664778 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.664782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.665272 (+ 490us) server_negotiation.cc:407] Connection header received
0806 04:44:14.665460 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.665463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.665508 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.665555 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.666006 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37396: BlockingRecv error: recv got EOF from 127.0.0.1:37396 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260806 04:44:14.673734 24292 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:14.672160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.31.166.189:43629)
0806 04:44:14.672308 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.672312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.672801 (+ 489us) server_negotiation.cc:407] Connection header received
0806 04:44:14.672989 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.672994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.673043 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.673108 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.673645 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37408: BlockingRecv error: recv got EOF from 127.0.0.1:37408 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 04:44:14.685837 24292 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:14.684132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.31.166.189:43629)
0806 04:44:14.684252 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.684257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.684834 (+ 577us) server_negotiation.cc:407] Connection header received
0806 04:44:14.685042 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.685046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.685101 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.685154 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.685720 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37424: BlockingRecv error: recv got EOF from 127.0.0.1:37424 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260806 04:44:14.706681 24292 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:14.704572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.31.166.189:43629)
0806 04:44:14.704704 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.704710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.705549 (+ 839us) server_negotiation.cc:407] Connection header received
0806 04:44:14.705788 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.705792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.705855 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.705915 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.706552 (+ 637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37436: BlockingRecv error: recv got EOF from 127.0.0.1:37436 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 04:44:14.742617 24292 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:14.741405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.31.166.189:43629)
0806 04:44:14.741570 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.741575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.741696 (+ 121us) server_negotiation.cc:407] Connection header received
0806 04:44:14.741895 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.741897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.741946 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.742040 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.742507 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37452: BlockingRecv error: recv got EOF from 127.0.0.1:37452 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 04:44:14.810411 24292 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:14.808992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37454 (local address 127.31.166.189:43629)
0806 04:44:14.809144 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.809148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.809398 (+ 250us) server_negotiation.cc:407] Connection header received
0806 04:44:14.809543 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.809546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.809596 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.809678 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.810270 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37454: BlockingRecv error: recv got EOF from 127.0.0.1:37454 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 04:44:14.942291 24292 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:14.941025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.31.166.189:43629)
0806 04:44:14.941191 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:14.941196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:14.941397 (+ 201us) server_negotiation.cc:407] Connection header received
0806 04:44:14.941556 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:14.941558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:14.941611 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:14.941686 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:14.942133 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37456: BlockingRecv error: recv got EOF from 127.0.0.1:37456 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 04:44:15.201934 24292 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 04:44:15.200624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.31.166.189:43629)
0806 04:44:15.200807 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:15.200811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:15.200971 (+ 160us) server_negotiation.cc:407] Connection header received
0806 04:44:15.201171 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:15.201175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:15.201224 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:15.201317 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 04:44:15.201798 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37470: BlockingRecv error: recv got EOF from 127.0.0.1:37470 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 04:44:15.500126 24400 catalog_manager.cc:1259] Loaded cluster ID: 8fa87a362cb14c8d9b381e9bd246ffd3
I20260806 04:44:15.500213 24400 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc: loading cluster ID for follower catalog manager: success
I20260806 04:44:15.501868 24400 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc: acquiring CA information for follower catalog manager: success
I20260806 04:44:15.502969 24400 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fa435521895a4fbabd17557a22e8c8bc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:44:15.520277 24419 catalog_manager.cc:1259] Loaded cluster ID: 8fa87a362cb14c8d9b381e9bd246ffd3
I20260806 04:44:15.520378 24419 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507: loading cluster ID for follower catalog manager: success
I20260806 04:44:15.522388 24419 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507: acquiring CA information for follower catalog manager: success
I20260806 04:44:15.523694 24419 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af78723eb85c47aeafeb85fcc2ef3507: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 04:44:15.720669 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:15.716489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37474 (local address 127.31.166.189:43629)
0806 04:44:15.716787 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:15.716790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:15.716849 (+ 59us) server_negotiation.cc:407] Connection header received
0806 04:44:15.716990 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:15.716993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:15.717129 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:15.717291 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:44:15.717580 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:15.718117 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:15.718981 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:15.719322 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:15.719410 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:15.719708 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:44:15.720139 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:44:15.720234 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:15.720405 (+ 171us) server_negotiation.cc:299] Negotiation successful
0806 04:44:15.720466 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":75,"threads_started":1}
I20260806 04:44:15.727526 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:15.723863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.31.166.188:33477)
0806 04:44:15.724129 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:15.724133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:15.724147 (+ 14us) server_negotiation.cc:407] Connection header received
0806 04:44:15.724271 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:15.724274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:15.724391 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:15.724490 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:44:15.724672 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:15.725182 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:15.726017 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:15.726242 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:15.726323 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:15.726609 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:44:15.727016 (+ 407us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:44:15.727177 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:15.727260 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 04:44:15.727317 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":81,"threads_started":1}
I20260806 04:44:15.734632 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:15.730654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47796 (local address 127.31.166.188:33477)
0806 04:44:15.730763 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:15.730768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:15.730990 (+ 222us) server_negotiation.cc:407] Connection header received
0806 04:44:15.731206 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:15.731212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:15.731265 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:15.731328 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:44:15.731637 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:15.732435 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:15.733433 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:15.733747 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:15.733859 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:15.734149 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:44:15.734239 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:44:15.734319 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:15.734430 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 04:44:15.734480 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 04:44:15.735004 24501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:15.730825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.31.166.189:43629)
0806 04:44:15.730952 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:15.730958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:15.731221 (+ 263us) server_negotiation.cc:407] Connection header received
0806 04:44:15.731411 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:15.731415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:15.731468 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:15.731522 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:44:15.731916 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:15.732642 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:15.733722 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:15.734047 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:15.734149 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:15.734494 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:44:15.734566 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:44:15.734610 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:15.734797 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 04:44:15.734860 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 04:44:15.735386 24514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 04:44:15.730545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34636 (local address 127.31.166.190:40581)
0806 04:44:15.731153 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0806 04:44:15.731158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 04:44:15.731177 (+ 19us) server_negotiation.cc:407] Connection header received
0806 04:44:15.731274 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 04:44:15.731277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 04:44:15.731436 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 04:44:15.731557 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 04:44:15.731810 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:15.732367 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:15.733497 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 04:44:15.733844 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 04:44:15.733975 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 04:44:15.734330 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 04:44:15.734659 (+ 329us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 04:44:15.735130 (+ 471us) server_negotiation.cc:1035] Waiting for connection context
0806 04:44:15.735171 (+ 41us) server_negotiation.cc:299] Negotiation successful
0806 04:44:15.735239 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":100,"threads_started":1}
I20260806 04:44:15.738857 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 24134
I20260806 04:44:15.746879 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 24213
I20260806 04:44:15.753639 32410 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskk51eOb/build/debug/bin/kudu with pid 24293
2026-08-06T04:44:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3144 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6264 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (295607 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST